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Abstract 


We consider the problem of developing robust algorithms which cope with noisy data. In the 
Probably Approximately Correct model of machine learning, we develop a general technique 
which allows nearly all PAC learning algorithms to be converted into highly efficient PAC 
learning algorithms which tolerate noise. In the field of combinatorial algorithms, we develop 
techniques for constructing search algorithms which tolerate linearly bounded errors and prob- 
abilistic errors. 

In the field of machine learning, we derive general bounds on the complexity of learning in 
the recently introduced Statistical Query model and in the PAC model with noise. We do so 
by considering the problem of improving the accuracy of learning algorithms. In particular, we 
study the problem of “boosting” the accuracy of “weak” learning algorithms which fall within 
the Statistical Query model, and we show that it is possible to improve the accuracy of such 
learning algorithms to any arbitrary accuracy. We derive a number of interesting consequences 
from this result, and in particular, we show that nearly all PAC learning algorithms can be 
converted into highly efficient PAC learning algorithms which tolerate classification noise and 
malicious errors. 

We also investigate the longstanding problem of searching in the presence of errors. We 
consider the problem of determining an unknown quantity « by asking “yes-no” questions, 
where some of the answers may be erroneous. We focus on two different models of error: 
the linearly bounded model, where for some known constant r < 1/2, each initial sequence of 
t answers is guaranteed to have no more than ri errors, and the probabilistic model, where 
errors occur randomly and independently with probability p < 1/2. We develop highly efficient 
algorithms for searching in the presence of linearly bounded errors, and we further show that 
searching in the presence of probabilistic errors can be efficiently reduced to searching in the 
presence of linearly bounded errors. 
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CHAPTER 1 


Introduction 


This thesis is concerned with the problems of concept learning and searching, and in particular, 
it deals with the problem of coping with faulty data in each of these settings. In the sections 


that follow, we informally introduce the topics of concept learning and searching. 


1.1 Concept Learning 


A concept is simply a rule which divides objects into two categories: positive examples and 
negative examples. The concept “circular region of radius 1 centered at the origin,” for instance, 
divides all points in the plane into positive and negative examples. All points in the plane 
whose Euclidean distance to the origin is at most 1 are positive examples of this concept, while 
all points in the plane whose Euclidean distance to the origin is greater than 1 are negative 
examples. Concept learning is the problem of inferring a rule from a set of positive and negative 
examples of that rule. 

Devising machines, algorithms and programs which can learn concepts from positive and 
negative examples is an important goal of artificial intelligence, and it has motivated much 
research in the field. In this thesis, we approach the problem of concept learning from the 
perspective of computational learning theory [1] in that we are concerned with the ability to 
learn concepts efficiently. A machine or algorithm for concept learning is said to be efficient if 


the quantities of resources it uses (e.g. time, space, examples, etc.) are bounded by polynomials 


11 


12 Introduction 


in the various learning parameters. 

Many models of concept learning have been developed, and we focus on one such model, the 
Probably Approximately Correct (PAC) model introduced by Valiant [34]. In the PAC model 
of learning, the unknown concept to be learned is assumed to be a member of a known concept 
class, and our goal is to develop a general algorithm which can learn any concept in the known 
class from examples of that concept. In the above example, for instance, the known concept 


> and the unknown target concept is “circular region 


class may be “circular regions in the plane,’ 
of radius 1 centered at the origin.” 

A great many algorithms have been devised to “PAC-learn” various concept classes [1]; 
however, nearly all of these algorithms are “brittle” in the sense that they cannot tolerate noisy 
data. In Part I of this thesis, a model of PAC learning is studied which is both general in the 
sense that it encompasses nearly all known PAC learning algorithms and robust in the sense that 
many types of noise can be cleanly and efficiently accommodated. We extend and improve this 
model of learning in ways which both increase the power of the algorithm designer and decrease 


the complexity of the resultant robust learning algorithms. A more formal introduction to this 


topic and to the corresponding results contained within this thesis can be found in Chapter 2. 


1.2 Searching 


The problem of search is a fundamental one in computer science, and its importance need hardly 
be justified. Perhaps the simplest example of search is embodied in the problem of finding a 
particular element within a sorted collection of elements. The standard binary search algorithm 
can be used to optimally solve this problem wherein all of the requisite queries are comparison 
questions. This, however, is but one type of search. Consider also, for example, the problem 
of medical diagnosis. A patient is suffering from some unknown disease, and a doctor has at 
his disposal a number of tests, each of which rules out some number of possible maladies. By 
performing some number of these tests, the doctor, can, in principle, eliminate all possibilities 
until only one remains. This too is an example of search wherein the “elements” (possible 
diseases) are not ordered and the queries themselves correspond to subset questions. Instances 
of search are pervasive, and we provide these two examples merely for illustrative purposes. 


The problem of searching in an optimal fashion is well understood, but not if one were to 
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allow “faulty” data. For example, consider the aforementioned medical diagnosis example, and 
suppose that some tests were known to occasionally be false-positive or false-negative. It is 
problems such as this which motivate the work in Part II of this thesis wherein various models 
of searching in the presence of faulty data are studied and robust algorithms for performing 
search are developed. A more formal introduction to this topic and to the corresponding results 


contained within this thesis can be found in Chapter 7. 


Part I 


Learning in the Presence of Noise 
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CHAPTER 2 


Introduction 


The statistical query model of learning was created so that algorithm designers could construct 
noise-tolerant PAC learning algorithms in a natural way. Ideally, such a model of robust learning 
should restrict the algorithm designer as little as possible while maintaining the property that 
these new learning algorithms can be efficiently simulated in the PAC model with noise. In 
the following chapters, we both extend and improve the current statistical query model in 
ways which both increase the power of the algorithm designer and decrease the complexity 
of simulating these new learning algorithms. In this chapter, we summarize our results and 


introduce the various models of learning required for the exposition that follows. 


2.1 Introduction 


Since Valiant’s introduction of the Probably Approximately Correct model of learning [34], 
PAC learning has proven to be an interesting and well studied model of machine learning. In 
an instance of PAC learning, a learner is given the task of determining a close approximation 
of an unknown {0,1}-valued target function f from labelled examples of that function. The 
learner is given access to an example oracle and accuracy and confidence parameters. When 
polled, the oracle draws an instance according to a distribution D and returns the instance 
along with its label according to f. The error rate of an hypothesis output by the learner is the 


probability that an instance chosen according to D will be mislabelled by the hypothesis. The 
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learner is required to output an hypothesis such that, with high confidence, the error rate of the 
hypothesis is less than the accuracy parameter. Two standard complexity measures studied in 
the PAC model are sample complexity and time complexity. Efficient PAC learning algorithms 
have been developed for many function classes [1], and PAC learning continues to be a popular 
model of machine learning. 

The model of learning described above is often referred to as the strong learning model since 
a learning algorithm may be required to output an arbitrarily accurate hypothesis depending on 
the accuracy parameter supplied. An interesting variant referred to as the weak learning model is 
identical, except that there is no accuracy parameter and the output hypothesis need only have 
error rate slightly less than 1/2. In other words, the output of a weak learning algorithm need 
only perform slightly better than random guessing. A fundamental and surprising result first 
shown by Schapire [28, 29] and later improved upon by Freund [14, 15] states that any algorithm 
which efficiently weakly learns can be transformed into an algorithm which efficiently strongly 
learns. These results have important consequences for PAC learning, including providing upper 
bounds on the time and sample complexities of strong learning. 

One criticism of the PAC model is that the data presented to the learner is assumed to 
be noise-free. In fact, most of the standard PAC learning algorithms would fail if even a 
small number of the labelled examples given to the learning algorithm were “noisy.” Two 
popular noise models for both theoretical and experimental research are the classification noise 
model introduced by Angluin and Laird [2, 21] and the malicious error model introduced by 
Valiant [35] and further studied by Kearns and Li [20]. In the classification noise model, each 
example received by the learner is mislabelled randomly and independently with some fixed 
probability. In the malicious error model, an adversary is allowed, with some fixed probability, 
to substitute a labelled example of his choosing for the labelled example the learner would 
ordinarily see. 

While a limited number of efficient PAC algorithms had been developed which tolerate 
classification noise [2, 16, 26], no general framework for efficient learning! in the presence of 
classification noise was known until Kearns introduced the Statistical Query model [19]. 


'Angluin and Laird [2] introduced a general framework for learning in the presence of classification noise. 
However, their methods do not yield computationally efficient algorithms in most cases. 
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In the SQ model, the example oracle of the standard PAC model is replaced by a statistics 
oracle. An $Q algorithm queries this new oracle for the values of various statistics on the 
distribution of labelled examples, and the oracle returns the requested statistics to within some 
specified additive error. Upon gathering a sufficient number of statistics, the 5Q algorithm 
returns an hypothesis of the desired accuracy. Since calls to a statistics oracle can be simulated 
with high probability by drawing a sufficiently large sample from an example oracle, one can 
view this new oracle as an intermediary which effectively limits the way in which a learning algo- 
rithm can make use of labelled examples. Two standard complexity measures of SQ algorithms 
are query complexity, the maximum number of statistics required, and tolerance, the minimum 
additive error required. The time and sample complexities of simulating 5Q algorithms in the 
PAC model are directly affected by these measures; therefore, we would like to bound these 
measures as closely as possible. 

Kearns [19] has demonstrated two important properties of the SQ model which make it wor- 
thy of study. First, he has shown that nearly every PAC learning algorithm can be cast within 
the 5Q model, thus demonstrating that the SQ model is quite general and imposes a rather 
weak restriction on learning algorithms. Second, he has shown that calls to a statistics oracle 
can be simulated, with high probability, by a procedure which draws a sufficiently large sample 
from a classification noise oracle. An immediate consequence of these two properties is that 
nearly every PAC learning algorithm can be transformed into one which tolerates classification 
noise. 

Decatur [9] has further demonstrated that calls to a statistics oracle can be simulated, with 
high probability, by a procedure which draws a sufficiently large sample from a malicious error 
oracle. Thus, nearly every PAC learning algorithm can be transformed into one which tolerates 
malicious errors. While Kearns and Li [20] had previously demonstrated a general technique 
for converting a PAC learning algorithm into one which tolerates small amounts of malicious 
error, the results obtained by appealing to SQ are better in some interesting cases [9]. 

While greatly expanding the function classes known to be learnable in the presence of noise, 
Kearns’ technique does not constitute a formal reduction from PAC learning to SQ learning. 
In fact, such a reduction cannot exist: while the class of parity functions is known to be PAC 


learnable [17], Kearns has shown that this class is provably unlearnable in the SQ model. 
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Kearns’ technique for converting PAC algorithms to 5Q algorithms consists of a few general 
tules, but each PAC algorithm must be examined in turn and converted to an 5Q algorithm 
individually. Thus, one cannot derive general upper bounds on the complexity of SQ learning 
from upper bounds on the complexity of PAC learning, due to the dependence on the specific 
conversion of a PAC algorithm to an SQ algorithm. A consequence of this fact is that general 
upper bounds on the time and sample complexities of PAC learning in the presence of noise are 
not directly obtainable either. 

We obtain bounds for SQ learning and PAC learning in the presence of noise by making 
use of the following result. We define weak SQ learning in a manner analogous to weak PAC 
learning, and we show that it is possible to boost the accuracy of weak SQ algorithms to obtain 
strong 5Q algorithms. Thus, we show that weak SQ learning is equivalent to strong 5Q learning. 
We use the technique of “boosting by majority” [15] which is nearly optimal in terms of its 
dependence on the accuracy parameter e€. 

In the SQ model, as in the PAC model, this boosting result allows us to derive general 
upper bounds on many complexity measures of learning. Specifically, we derive simultaneous 
upper bounds with respect to € on the number of queries, O(log” +), the Vapnik-Chervonenkis 
dimension of the query space, O(log + log log +), and the inverse of the minimum tolerance, 
O(4 log +). In addition, we show that these general upper bounds are nearly optimal by de- 


scribing a class of learning problems for which we simultaneously lower bound the number 


4 


of queries by Osea 


log +) and the inverse of the minimum tolerance by 0(+). Here d is the 
Vapnik-Chervonenkis dimension of the function class to be learned. 

The complexity of a statistical query algorithm in conjunction with the complexity of simu- 
lating SQ algorithms in the various noise models determine the complexity of the noise-tolerant 
PAC learning algorithms obtained. Kearns [19] has derived general bounds on the minimum 
complexity of SQ algorithms, and we derive some specific lower bounds as well. Our boosting 
result provides a general technique for constructing SQ algorithms which are nearly optimal 
with respect to these bounds. However, the robust PAC learning algorithms obtained by sim- 
ulating even optimal 5Q algorithms in the presence of noise are inefficient when compared to 


known lower bounds for PAC learning in the presence of noise [11, 20, 30]. In fact, the PAC 


learning algorithms obtained by simulating optimal 5Q algorithms in the absence of noise are 
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inefficient when compared to the tight bounds known for noise-free PAC learning [7, 11]. These 
shortcomings could be consequences of either inefficient simulations or a deficiency in the model 
itself. In this thesis, we show that both of these explanations are true, and we provide both 
new simulations and a variant of the SQ model which combat the current inefficiencies of PAC 
learning via the statistical query model. 

We improve the complexity of simulating $Q algorithms in the presence of classification 
noise by providing a more efficient simulation. If 7, is a lower bound on the minimum additive 
error requested by an SQ algorithm and 7, < 1/2 is an upper bound on the unknown noise 


rate, then Kearns’ original simulation essentially runs O( different copies of the SQ 


1 
ine) 
algorithm and processes the results of these runs to obtain an output. We show that this 


1 
1-2 


“branching factor” can be reduced to e(= log ), thus reducing the time complexity of the 
simulation. We also provide a new and simpler proof that statistical queries can be estimated in 
the presence of classification noise, and we show that our formulation can easily be generalized 
to accommodate a strictly larger class of statistical queries. 

We improve the complexity of simulating $Q algorithms in the absence of noise and in the 
presence of malicious errors by proposing a natural variant of the SQ model and providing 
efficient simulations for this variant. In the relative error SQ model, we allow SQ algorithms to 
submit statistical queries whose estimates are required within some specified relative error. We 
show that a class is learnable with relative error statistical queries if and only if it is learnable 
with (standard) additive error statistical queries. Thus, known learnability and hardness results 
for statistical query learning [6, 19] also hold in this variant. 

We demonstrate general bounds on the complexity of relative error SQ learning, and we 
show that many learning algorithms can naturally be written as highly efficient, relative error 
5Q algorithms. We further provide simulations of relative error SQ algorithms in both the 
absence and presence of noise. These simulations in the absence of noise and in the presence 
of malicious errors are more efficient than the simulations of additive error SQ algorithms, and 
given a roughly optimal relative error SQ algorithm, these simulations yield roughly optimal 
PAC learning algorithms. These results hold for all function classes which are SQ learnable 

Finally, we show that our simulations of SQ algorithms in the absence of noise, in the 


presence of classification noise, and in the presence malicious errors can all be modified to 
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accommodate a strictly larger class of statistical queries. In particular, we show that our simu- 
lations can accommodate real-valued statistical queries. Real-valued queries allow an algorithm 
to query the expected value of a real-valued function of labelled examples. Our results on 
improved simulations hold for this generalization in both the absence and presence of noise. 
The remainder of this work is organized as follows. In Section 2.2, we formally define the 
learning models of interest, and in Section 2.3, we describe PAC model boosting results which 
are used in later chapters. In Chapters 3 and 4, we present our additive error and relative error 
5Q model results, respectively. In Chapter 5, we present some extensions of our results, and 


we conclude with a discussion of some open questions in Chapter 6. 


2.2 Learning Models 


In this section, we formally define the relevant models of learning necessary for the exposition 
that follows. We begin by defining the weak and strong PAC learning models, followed by the 


classification noise and malicious error models, and finally the statistical query model. 


2.2.1 The Weak and Strong PAC Learning Models 


In an instance of PAC learning, a learner is given the task of determining a close approximation 
of an unknown {0,1}-valued target function from labelled examples of that function. The 
unknown target function f is assumed to be an element of a known function class F defined 
over an instance space X. The instance space X is typically either the Boolean hypercube 
{0,1}" or n-dimensional Euclidean space #”. We use the parameter n to denote the common 
length of each instance « € X. 

We assume that the instances are distributed according to some unknown probability dis- 
tribution D on X. The learner is given access to an example oracle EX(f, D) as its source of 
data. A call to EX(f, D) returns a labelled example (x,/) where the instance x € X is drawn 
randomly and independently according to the unknown distribution D, and the label / is equal 
to f(a). We often refer to a sequence of labelled examples drawn from an example oracle as a 
sample. 


A learning algorithm draws a sample from LX(f, D) and eventually outputs an hypothesis 
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h from some hypothesis class H defined over X. For any hypothesis h, the error rate of h is 
defined to be the probability that h mislabels an instance drawn randomly according to D. 
By using the notation Prp[P(«)] to denote the probability that a predicate P is satisfied by 
an instance drawn randomly according to D, we may define error(h) = Prp[h(2) # f(x)]. We 
often think of H as a class of representations of functions in F, and as such we define size(f) 
to be the size of the smallest representation in H of the target function /f. 

The learner’s goal is to output, with probability at least 1<6, an hypothesis h whose error 
rate is at most ¢, for the given accuracy parameter € and confidence parameter 6. A learning 
algorithm is said to be polynomially efficient if its running time is polynomial in 1/e, 1/6, 


and size(f). We formally define PAC learning as follows (adapted from Kearns [19]): 


Definition 1 (Strong PAC Learning) 

Let F and H be function classes defined over X. The class F is said to be polynomially learnable 
by H if there exists a learning algorithm A and a polynomial p(-,-,-,-) such that for any f € F, 
for any distribution D on X, for any accuracy parameter ¢, 0 <¢< 1, and for any confidence 
parameter 6,0 <6 <1, the following holds: if A is given inputs « and 6, and access to an 
example oracle EX(f,D), then A halts in time bounded by p(1/e,1/6,n, size(f)) and outputs 
an hypothesis h € H that with probability at least 1<6 satisfies error(h) < e. 


As stated, this is often referred to as strong learning since the learning algorithm may be 
required to output an arbitrarily accurate hypothesis depending on the input parameter «. A 
variant of strong learning called weak learning is identical, except that there is no accuracy 
parameter € and the output hypothesis need only have error rate slightly less than 1/2, i.e. 
error(h) < $o7 = 3 Pan for some polynomial p. Since random guessing would produce 
an error rate of 1/2, one can view the output of a weak learning algorithm as an hypothesis 
whose error rate is slightly better than random guessing. We refer to the output of a weak 


learning algorithm as a weak hypothesis and the output of a strong learning algorithm as a 


strong hypothesis. 


2.2.2 The Classification Noise and Malicious Error Models 


One criticism of the PAC model is that the data presented to the learner is required to be 


noise-free. Two popular models of noise for both experimental and theoretical purposes are 
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the classification noise model introduced by Angluin and Laird [2, 21] and the malicious error 


model introduced by Valiant [35]. 


The Classification Noise Model 


In the classification noise model, the example oracle EX(f, D) is replaced by a noisy example 
oracle EX%,(f, D). Each time this noisy example oracle is called, an instance x € X is drawn 
according to D. The oracle then outputs (2, f(#)) with probability 1 <7 or (#,3f(a)) with 
probability 7, randomly and independently for each instance drawn. Despite the noise in the 
labelled examples, the learner’s goal remains to output an hypothesis h which, with probability 
at least 1 <6, has error rate error(h) = Prp[h(x) # f(#)] at most e. 

While the learner does not typically know the exact value of the noise rate 7, the learner 
is given an upper bound 7 on the noise rate, 0 < 7 < m < 1/2, and the learner is said to be 
polynomially efficient if its running time is polynomial in the usual PAC learning parameters 


as well as =. 
—4Nb 


The Malicious Error Model 


In the malicious error model, the example oracle EX(f,D) is replaced by a noisy example 
oracle EX ea (fh D). When a labelled example is requested from this oracle, with probability 
1, an instance x is chosen according to D and (2, f(x)) is returned to the learner. With 
probability 3, a malicious adversary selects any instance x € X, selects a label J € {0,1}, and 
returns (#,/). Again, the learner’s goal is to output an hypothesis h which, with probability at 


least 1 <6, has error rate error(h) = Prp[h(z) # f(x)] at most e. 


2.2.3. The Statistical Query Model 


In the statistical query model, the example oracle EX(f, D) from the standard PAC model is 
replaced by a statistics oracle STAT(f, D). An SQ algorithm queries the STAT oracle for the 
values of various statistics on the distribution of labelled examples (e.g. “What is the probability 
that a randomly chosen labelled example (x,/) has variable x; = 0 and / = 1?”), and the STAT 
oracle returns the requested statistics to within some specified additive error. Formally, a 


statistical query is of the form [x,7]. Here y is a mapping from labelled examples to {0,1} (i.e. 
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x: X x {0,1} — {0,1}) corresponding to an indicator function for those labelled examples 
about which statistics are to be gathered, while 7 is an additive error parameter. A call [y,7] 
to STAT(f, D) returns an estimate P, of Py = Prp[y(a, f(”))] which satisfies |P, @P,| <r. 
A call to STAT(f,D) can be simulated, with high probability, by drawing a sufficiently 
large sample from FX(f,D) and outputting the fraction of labelled examples which satisfy 
x(«, f(z)) as the estimate Py. Since the required sample size depends polynomially on 1/7 and 
the simulation time additionally depends on the time required to evaluate y, an SQ learning 
algorithm is said to be polynomially efficient if 1/7, the time required to evaluate each y, and 
the running time of the SQ algorithm are all bounded by polynomials in 1/e, n and size(f). We 
formally define polynomially efficient learning in the statistical query model as follows (adapted 


from Kearns [19]): 


Definition 2 (Strong SQ Learning) 

Let F and H be function classes defined over X. The class F is said to be polynomially 
learnable via statistical queries by H if there exists a learning algorithm A and polynomials 
Pil-y+5+), Pal, +,+), and ps(-,-,-) such that for any f € F, for any distribution D on X, and for 
any error parameter ¢, 0 < ¢€ <1, the following holds: if A is given input € and access to a 
statistics oracle STAT(f, D), then (1) for every query [x,7] made by A, x can be evaluated in 
time bounded by pi(1/e,n, size(f)) and 1/7 is bounded by po(1/e,n, size( f)), and (2) A halts in 
time bounded by p3(1/e,n, size(f)) and outputs an hypothesis h € H that satisfies error(h) < e. 


For an SQ algorithm A, we may further define its query complexity and tolerance. In a given 
instance of learning, the query complexity of A is the number of queries submitted by A, and 
the tolerance of A is the smallest additive error requested by A. We let NV, = N.(€,n, size( f)) 
be an upper bound on the query complexity of A, and we let 7, = 7,(€,n, size(f)) be a lower 
bound on the tolerance of A. Note that N. < p3(1/e,n, size(f)) and 7, > 1/po(1/e, n, size(f)). 

Since calls to a statistics oracle can be simulated by a procedure which draws a sample from 
an example oracle, one can view the statistical query model as simply restricting the way in 
which PAC learning algorithms can make use of labelled examples. Kearns has shown that this 
restriction is rather weak in that nearly every PAC learning algorithm can be cast in the SQ 


model. 
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An important property of this model is that calls to a statistics oracle can also be simulated, 
with high probability, by a procedure which draws a sample from a classification noise oracle 
EX’..(f, D) [19] or a malicious error oracle EX%,,,(f,D) [9]. In the former case, the sample 
size required is polynomial in 1/7, 1/(1 2m) and log(1/6); in the latter case, the sample 
size required is polynomial in 1/7 and log(1/é). While a reasonably efficient simulation of 
an 5Q algorithm can be obtained by drawing a separate sample for each call to the statistics 
oracle, better bounds on the sample complexity of the simulation are obtained by drawing one 
large sample and estimating each statistical query using that single sample. If we let Q be the 
function space from which an SQ algorithm A selects its queries, then the size of the single 
sample required is independent of the query complexity of A but depends on either the size of 
Q or the Vapnik-Chervonenkis dimension? of Q. Q is referred to as the query space of the SQ 
algorithm A. 

Kearns has shown that an SQ algorithm can be simulated in the classification noise model 
using a sample size which depends on Q, 7,, 6, € and ». Decatur has shown that an SQ 
algorithm can be simulated in the malicious error model using a sample size which depends on 
QO, 7, and 6. The amount of malicious error which can be tolerated by the latter simulation 
depends on 7,. Given that nearly every PAC learning algorithm can be converted to an SQ 
algorithm, an immediate consequence of these results is that nearly every PAC algorithm can 
be transformed into one which tolerates noise. The complexities of these noise-tolerant versions 
depend on 7, and Q, which themselves are a function of the ad hoc conversion of PAC algorithms 
to $Q algorithms. Thus, one cannot show general upper bounds on the complexity of these 
noise-tolerant versions of converted PAC algorithms. 

We define weak SQ learning identically to strong SQ learning except that there is no accuracy 
parameter e. In this case, the output hypothesis need only have error rate slightly less than 1/2, 
i.e. error(h) < Soy = 3 Pana for some polynomial p. By showing that weak SQ learning 
algorithms can be “boosted” to strong SQ learning algorithms, we derive general lower bounds 
on the tolerance of 5Q learning and general upper bounds on the complexity of the requisite 
query space. We are then able to show general upper bounds on the complexity noise-tolerant 


PAC learning via the statistical query model. These results are given in Chapters 3 and 4. 


?VC-dimension is a standard complexity measure for a space of {0,1}-valued functions. 
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2.3. Boosting in the PAC Model 


In this section, we describe the PAC model boosting results on which our SQ model boosting 
results are based. 

Schapire [28, 29] and Freund [14, 15] use similar strategies for boosting weak learning al- 
gorithms to strong learning algorithms. They both create a strong hypothesis by combining 
many hypotheses obtained from multiple runs of a weak learning algorithm. The boosting 
schemes derive their power by essentially forcing the weak learning algorithm, in later runs, 
to approximate the target function f with respect to new distributions which “heavily” weight 
those instances that previous hypotheses misclassify. By suitably constructing example oracles 
corresponding to these new distributions and properly combining the hypotheses obtained from 
multiple runs of the weak learning algorithm, a strong learning algorithm can be produced 
which uses the weak learning algorithm as a subroutine. 

Freund has developed two similar methods (which we call Scheme 1 and Scheme 2) for 
boosting weak learning algorithms to strong learning algorithms. One is more efficient with 
respect to € while the other is more efficient with respect to y. Freund develops a hybrid 
scheme more efficient than either Scheme 1 or Scheme 2 by combining these two methods in 
order to capitalize on the advantages of each. We first describe the two schemes separately and 


then show how to combine them. 


2.3.1 Boosting via Scheme 1 in the PAC Model 


Scheme 1 uses a weak learning algorithm to create a set of k, = aed In + 


weak hypotheses and 
outputs the majority vote of these hypotheses as the strong hypothesis. The weak hypotheses 
are created by asking the weak learner to approximate f with respect to various modified distri- 
butions over the instance space X. The distribution used to generate a given weak hypothesis 
is based on the performance of the previously generated weak hypotheses. Hypothesis h, is 


created in the usual way by using FX(f,D). For all i > 1, hypothesis h;,, is created by giving 


the weak learner access to a filtered example oracle EF. X(f, Dj41) defined as follows: 
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. Draw a labelled example (x, f(x)) from E.X(f, D). 
. Compute hi (a),..., hg(x). 


. Set r to be the number of hypotheses which agree with f on z. 


. Flip a biased coin with Pr[HEap] = at. 


. If HEAD, then output example (x, f(#)), otherwise go to Step 1. 


When k weak hypotheses are to be generated, the set of probabilities {a‘} are fixed according 


to the following binomial distribution: 


0 ifr> [4 
(gd Gt Hel Gg eel 

ifiel[S]+i<r< |F| 

0 ifr<ielfe]tl 

Freund shows that, with high probability, the majority vote of h1,...,h,, has error rate no 
more than € with respect to D if each h; has error rate no more than ; <7 with respect to D;. 

One pitfall of this scheme is that the simulation of EX(f,D;4,) may need to draw many 
examples from EX(f,D) before one is output to the weak learner. Let t; be the probability 
that an example drawn randomly from E-X(f,D) passes through the probabilistic filter which 
defines PX(f, D4). Freund observes that if t; < ce? for some constant c, then the majority 
vote of hy,...,h; is already a strong hypothesis. The boosting algorithm can estimate ¢;, and if 
t; is below the cutoff, the algorithm may halt and output the majority vote of the hypotheses 
created thus far. The boosting algorithm’s time and sample complexity dependence on y is 


O(1/y?), while its dependence on € is O(1/e?).3 


2.3.2 Boosting via Scheme 2 in the PAC Model 


Scheme 2 is very similar to Scheme 1. The weak learner is again called many times to provide 


weak hypotheses with respect to filtered distributions. This method uses ky = 2k, = saint 


“For asymptotically growing functions g, g > 1, we define O(g) to mean O(g log g) for some constant c > 0. 
For asymptotically shrinking functions g, 0 < g < 1, we define O(g) to mean O(g log°(1/g)) for some constant 
c > 0. We define 9 similarly for constants c < 0. Finally, we define © to mean both O and 9. This asymptotic 
notation, read “soft-O,” “soft-Omega,” and “soft-Theta,” is convenient for expressing bounds while ignoring 
lower order factors. It is somewhat different than the standard soft-order notation. 
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weak hypotheses, while the filtered example oracle remains the same. The main difference is the 
observation that if t; < at then we may simply use a “fair coin” in place of hj, and still 
be guaranteed, with high probability, that the final majority of ky hypotheses has error rate no 
more than ¢.*4 The boosting algorithm estimates ¢; to see if it is below this new threshold. If so, 
a “fair coin” is used as hypothesis A;4,, and the algorithm proceeds to find a weak hypothesis 


with respect to the next distribution. The boosting algorithm’s time and sample complexity 


dependence on y is O(1/7°), while its dependence on € is O(1/e). 


2.3.3. Hybrid Boosting in the PAC Model 


An improvement on these two boosting schemes is realized by using each in the “boosting 
range” for which it is most efficient. The first method is more efficient in 1/7, while the second 
method is more efficient in 1/e. We therefore use the first method to boost from $y to a 
constant, and we use the second method to boost from that constant to «. Let A1 be a learning 
algorithm which uses Scheme 1 and makes calls to the weak learning algorithm Ai_,. The 
strong learning algorithm A, uses Scheme 2 and makes calls to As as its “weak learner.” The 
strong hypothesis output by such a hybrid algorithm is a depth two circuit with a majority 
gate at the top level. The inputs to the top level are “fair coin” hypotheses and majority gates 
whose inputs are weak hypotheses with respect to various distributions. The hybrid’s time and 


sample complexity dependence on + is Q(1/7”), while its dependence on € is O(1/e). 


“A “fair coin” hypothesis ignores its input x and outputs the outcome of a fair coin flip. 


CHAPTER 8 


Learning Results in the 
Additive Error SQ Model 


In this chapter, we derive a number of results in the additive error statistical query model. 
We begin by showing that it possible to boost weak learning algorithms in the SQ model, 
and from this we derive general bounds on learning in the 5Q model. We then describe a 
new method for simulating $Q algorithms in the PAC model with classification noise. Finally, 
by combining the aforementioned results, we derive general bounds on PAC learning in the 


presence of classification noise which apply to all function classes known to be $Q learnable. 


3.1 Boosting in the Statistical Query Model 


Boosting is accomplished by forcing a weak learning algorithm to approximate the target func- 
tion f with respect to modified distributions over the instance space. Specifically, the boosting 
methods described in the previous chapter are based on the observation that, with high probabil- 
ity, the majority vote of h,,...,, has error rate at most € with respect to D if each constituent 
h; has error rate at most ; 7 with respect to D;. In the PAC model, a learner interacts with 
the distribution over the instance space through calls to an example oracle. Therefore, boosting 
in the PAC model is accomplished by constructing PX (f,D,;) from the original example oracle 
EX(f,D). In the SQ model, a learner interacts with the distribution over labelled examples 


through calls to a statistics oracle. Therefore, boosting in the 5Q model is accomplished by 
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constructing STAT(f, D;) from the original statistics oracle STAT(f, D). 

In the sections that follow, we first show how to boost a weak 5Q algorithm using either 
Scheme 1 or Scheme 2. We then show how to boost a weak SQ algorithm using the hybrid 
method. Although it is possible to boost in the SQ model using Schapire’s method, we do not 


describe these results since they are somewhat weaker than those presented here. 


3.1.1 Boosting via Scheme 1 in the Statistical Query Model 


We can use Scheme | to boost weak SQ learning algorithms by simply answering statistical 
queries made with respect to modified distributions. Therefore, we must be able to simulate 
queries to STAT(f, D;) by making queries to STAT(f,D). We first show how to specify the 
exact value of a query with respect to D; in terms of queries with respect to D. We then 
determine the accuracy with which we need to make these queries with respect to D in order 
to obtain a sufficient accuracy with respect to D;. 

The modified distributions required for boosting are embodied in the five step description 
of the filtered example oracle given in Section 2.3.1. Note that Steps 2 and 3 partition the 
instance space into 2+ 1 regions corresponding to those instances which are correctly classified 
by the same number of hypotheses. Let X' C X be the set of instances which are correctly 
classified by exactly r of the t hypotheses. We define the induced distribution Dz on a set Z with 
respect to distribution D as follows: For any Y C Z, Dz[Y] = D[Y]/D[Z]. By construction, 
for any given X? region, the filtered example oracle uniformly scales the probability with which 
examples from that region are drawn. Therefore, the induced distribution on X? with respect 
to D;41 is the same as the induced distribution on X/ with respect to D. (This fact is used to 
obtain Equation 3.2 from Equation 3.1 below.) 

A query [y,7] to STAT(f, Dj41) is a call for an estimate of Prp,,,[x(“, f(«))] within additive 


error T. We derive an expression for Prp,,,[x(a, f())] below. 


Prov lx(@, f(@))] = Yo Proalnles fee € X;)]-Prp,,,[ € X;] (3.1) 


= Yo Proix(e, fey lle € X")]-Pro.le € X;] (3.2) 
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_ so Peviale fey Ate Xz) op Prole € XY 
oe Prp[z € X?] Yi a’, - Prp[a € X5] 


_ = 0 ar, -Prolx (x , f(x)) A (x € X})] (3.3) 
Yio a’, - Prp[x € X3] 


Note that the denominator of Equation 3.3 is the probability that an example drawn randomly 
from E.X(f, D) passes through the probabilistic filter which defines FX (f, D;41). Recall that 
Freund calls this probability ¢;. 

Ignoring the additive error parameter for the moment, the probabilities in Equation 3.3 can 


be stated as queries to STAT(f, D) as follows 


vie 0% -STAT(f, D)[x A x5] 


STAT(f, Diat)[X] = yo a - STATUS, D)[X'] 


(3.4) 


where x} (#0) is true if and only if # € X}. Note that query xj is polynomially evaluatable 
given h,,...,h;, thus satisfying the efficiency condition given in the definition of SQ learning. 
We next determine the accuracy with which we must ask these queries so that the final 


result is within the desired additive error 7. We make use of the following two claims. 


Claim 1 /f 0 < a,b,¢,7 < 1 and a = b/c, then to obtain an estimate of a within additive 


error T, it is sufficient to obtain estimates of b and c within additive error cr /3. 


Proof: We must show that (6 + cr/3)/(¢ @c7/3) < a+7 and (6 Sc7/3)/(e+ c7T/3) > as. 


The claim is proven as follows. 


b+cr7/3 a+7/3 


cect/3 1sr/3 


= (a+7/3) (1 + +) 


= (a4+7/3)(14+7/2) 
= atar/2+7/34+7°/6 


< a4+T 
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beet/3 a &T/3 
et+er/3 1417/3 
= (a7/3) (1 ot.) 


> (a67/3)\1 47/3) 
= asat/3er/34+77/9 


> aST 


Claim 2 If 0 < s,p;,2%,7 < 1,0 < 30; p; < 1 and s = 3°; p;z;, then to obtain an estimate of 


8s within additive error 7, it is sufficient to obtain estimates of each z; within additive error T 


provided that the p; coefficients are known. 
Proof: The claim follows immediately from the inequalities given below. 
i Pla tT) = patty pe Sst 


i Pili OT) = Oy pi STL, pi = 8 ST 


Applying Claims 1 and 2 to Equation 3.4, we find that it is sufficient to submit queries 
to STAT(f, D) with additive error t; - 7/3 in order to simulate a call to STAT(f, Dj41) with 
additive error 7. There are two problems with this observation. First, if ¢; is small, then we 
are forced to submit queries with small additive error. Second, the value 7¢; is unknown, and in 
fact, it is the value of the denominator we are attempting to estimate. We can overcome these 
difficulties by employing the “abort” condition of Freund which allows us to either lower bound 
t; or abort the search for hj44. 

If t; < ce’, then the majority vote of the hypotheses generated thus far is a strong hypothesis. 


Submit each query to STAT(f,D) with additive error Let ¢; be the estimate for ¢; 


ce? 
243/7° 


obtained, and note that by Claim 2, f; is within additive error ss of t;. If t; < cP (1 Sas) 


then t; < ce?. In this case, we may halt and output the majority vote of the hypotheses 


created thus far. If ¢; > ce?(1 #aga7)> then t; > ce’ (1 Sagar) = cet). In this case, our 


estimate ¢, is sufficiently accurate since the additive error required by Claim 1 is t; - 7/3, and 
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t;-7T/3> ce( sah) -T/3= oF which is the additive error used. Given that the numerator 
and denominator are both estimated with additive error t; -7/3, their ratio is within additive 
error T by Claim 1. 

We can now bound the tolerance of strong SQ learning algorithms obtained by Scheme 1 
boosting. If tm = T(n, size(f)) is a lower bound on the tolerance of a weak SQ learning 
algorithm, then Q(79€7) is a lower bound on the tolerance of the strong SQ learning algorithm 
obtained by Scheme 1 boosting. 

We next examine the query complexity of strong SQ learning algorithms obtained by 
Scheme 1 boosting. Let No = No(n,size(f)) be an upper bound on the query complexity 
of a weak learner. In Equation 3.4, we note that 2(7+ 1) queries to STAT(f, D) are required to 
simulate a single query to STAT(f, Di41). Since ky = = In + is an upper bound on the number 
of weak learners run in the boosting scheme, O( Nok,”) = O(N log® +) is an upper bound on 
the query complexity of the strong 5Q learning algorithm obtained by Scheme 1 boosting. 

We finally examine the query space complexity of strong SQ learning algorithms obtained 
by Scheme 1 boosting. There are two cases to consider depending on the nature of the instance 
space. If the instance space is discrete, e.g. the Boolean hypercube {0,1}”, then the query space 
and hypothesis class used by an 5Q algorithm are generally finite. In this case, we can bound 
the size of the query space used by the strong SQ learning algorithm obtained by boosting, 
and this result is given below. If the instance space is continuous, e.g. n-dimensional Euclidean 
space Wt”, then the query space and hypothesis class used by an SQ algorithm are generally 
infinite. In this case, we can bound the VC-dimension of the query space used by the strong 
5Q learning algorithm obtained by boosting, and this result is given in the appendix. 

Let Q) and Ho be the finite query space and finite hypothesis class used by a weak SQ 
learning algorithm. The queries used by the strong 5Q learning algorithm obtained by Scheme 1 
boosting are of the form y, xj and XA; where y € Q, and xj is constructed from hypotheses in 
Ho. The queries xi are defined by ¢ hypotheses and a number 7,0 < 7 < 2. Since the hypotheses 
need not be distinct, for fixed 7 and j, the number of unique xj queries is (Il 421) - For fixed i, 
ol ti 1) 


the number of xj queries is (¢ + 1)- ( . Since 2 is bounded by &,, the total number of 


'This expression corresponds to the number of unique arrangements of i indistinguishable balls in |Ho| bins. 
Each unique arrangement corresponds to a unique x; in that the number of balls in bin € corresponds to the 
number of copies of the hypothesis associated with bin € used in yj. 
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xi queries is given by S772,(¢ + 1)- ("el"). Given that y € Qo, we may bound the size of 
the query space used by the strong $Q learning algorithm obtained from Scheme 1| boosting as 


follows: 


ky ky 
Qn] = |Qol + YUE + 1) PIP) + [Qo] DEE + 1) POP) 
t=1 


i=1 


In the appendix, it is shown that this expression has the following closed form: 
|Qx| = [Qo] + 1) (MP) + [Aol Qol + 1) CR) el 


Furthermore, it is shown the |Q,| is bounded above as follows: 
|Qn| < 2(|Qo] + 1)(|Hol + 2)” 


The complexity of simulating such an SQ algorithm in the various PAC models will depend on 
log |Qz|. We note that log |Q,| = O(log |Qo| + fy log |Ho|). 

Finally, in the appendix it is shown that the VC-dimension of the query space is bounded 
as follows: 


VC(Qp) = O( VC(Qo) + VC(Ho) « ky log k1) 


Theorem 1 Given a weak SQ learning algorithm whose query complexity is upper bounded by 
No = No(n, size(f)), whose tolerance is lower bounded by T) = To(n, size(f)), whose query space 
and hypothesis class are Qo and ‘Ho, respectively, and whose output hypothesis has error rate at 


most 5 O71; then a strong SQ learning algorithm can be constructed whose query complexity is 


O( No log* +) and whose tolerance is Q(t €’). The query space complexity is given by 


log |Qz| = O(log |Qo] + ss log = log |Ho]) 
when Qy and Hoy are finite, or 
VC(Qz) = O( VC(Qo) + VC(Ho) - (=z log z) log( <5 log +)) 


when Q, and Ho have finite VC-dimension. 


3.1.2 Boosting via Scheme 2 in the Statistical Query Model 


We can use Scheme 2 to boost weak SQ learning algorithms in a manner quite similar to that 


described above. Since the “abort” condition of Scheme 2 introduces “fair coin” hypotheses, we 
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first rederive the probability that y(2, f()) is true with respect to D;4, in terms of probabilities 
with respect to D. 

When 2 hypotheses have been generated, let w be the number of weak hypotheses and 
let « <w be the number of “fair coin” hypotheses. The weak hypotheses h,,...,h, partition 
the instance space X into w+ 1 regions corresponding to those instances which are correctly 
classified by the same number of weak hypotheses. Let XY C X be the set of instances which 
are correctly classified by exactly r of the w weak hypotheses. Consider the probability that an 
instance  € X passes through the probabilistic filter which defines FX (f, Di41). If none of the 
“fair coin” hypotheses agree with f, then this probability is a’. If 7 of the “fair coin” hypotheses 
agree with f, the this probability is ON js The total probability is thus A” = i =) 4 BF ” 
where ge = or 7) /2'-” is the probability that exactly 7 of the “fair coin” hypotheses agree 
with f. The following filtered example oracle is equivalent to EX(f, Di41): 


. Draw a labelled example (x, f(x)) from E.X(f, D). 
. Compute hy(a),..., hw(x). 


. Set r to be the number of hypotheses which agree with f on z. 


. Flip a biased coin with Pr[HEAD] = AX. 


. If HEAD, then output example (x, f(#)), otherwise go to Step 1. 


We may now derive an expression for Prp,,,[x(a, f(7))] as before. 


Probl Se) = LProuble Se) € XP): Proale €X2 


= So Proly(e, fe))h(e € X)]-Prp,,,[7 € X""] 


7 3 Prolx(a, f(@)) A (we XP )) AY Pro[e € X7"] 
7 Prp[a# € X”] yao AY + Prpla € X}] 


— peje Proll fey) Mee Xe) 
~ dyno AY + Prp[e € X}"] (3.5) 


— Vyao AP STAT PS, DD A x7] 
STAT(f, Disi)[x] = “SP STAT DIG) (3.6) 


Note that the denominators of Equations 3.5 and 3.6 again correspond to the probability ¢;. 


Also note that Sv", Av = S74 Dito ai, 3," <1 since the unique terms of the latter sum 
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are all contained in the product (S“‘_, a’ (is 5B )=1 

Applying Claims 1 and 2 to Equation 3.6, we again find that it is sufficient to submit queries 
to STAT(f, D) with additive error t; -7/3 in order to simulate a call to STAT(f, Dj41) with 
additive error 7. Again, there are two problems with this observation. First, if t; is small, then 
we are forced to submit queries with small additive error. Second, the value ¢; is unknown, and 
in fact, it is the value of the denominator we are attempting to estimate. We can overcome 
these difficulties by employing the “abort” condition of Freund which allows us to either lower 
bound ¢; or use a “fair coin” in place of hj41. 

If t; < 1 ¢e)y/In(1/e), then a “fair coin” can be used in place of h;4,. Submit each query 
to STAT(f, D) with additive error Canta Let ¢; be the estimate for the t; obtained, and 


note that by Claim 2, ¢; is within additive error sappeee of t;. If i; < atl S 


then t; < e(1¢)y/In(1/c). In this case, we may use a “fair coin” in place h;,, and proceed to 


e(l—e e(l—e _ e(l-e 3/7 
the next distribution. If f; > aaa Saga)» then ¢; > aie Sagar) = at ee): In 


this case, our estimate ¢, is sufficiently accurate since the additive error required by Claim 1 is 


t;-7/3, and t;-7/3 > Sass) T/3= Sapper which is the additive error used. Given 


that the numerator and denominator are both estimated with additive error ¢;-7/3, their ratio 


1 
aan) 


is within additive error 7 by Claim 1. 

We can now bound the tolerance of strong SQ learning algorithms obtained by Scheme 2 
boosting. If t|) = 7(n,size(f)) is a lower bound on the tolerance of a weak SQ learning 
algorithm, then Q(7,€7/ log(1/e)) is a lower bound on the tolerance of the strong SQ learning 
algorithm obtained by Scheme 2 boosting. 

We next examine the query complexity of strong SQ learning algorithms obtained by 
Scheme 2 boosting. Let No = No(n,size(f)) be an upper bound on the query complexity 
of a weak learner. In Equation 3.6, we note that 2(w+1) < 2(i+1) queries to STAT(f, D) are 
required to simulate a single query to STAT(f, Di41). Since ky = a3 In + is an upper bound on 
the number of weak learners run in the boosting scheme, O( Noks”) = O( Nox log” +) is an up- 
per bound on the query complexity of the strong 5Q learning algorithm obtained by Scheme 2 
boosting. 


We finally note that the query space complexity results for Scheme 2 boosting are identical 


to those for Scheme 1 boosting when k, is replace by ko. 
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Theorem 2 Given a weak SQ learning algorithm whose query complexity is upper bounded by 
No = No(n, size(f)), whose tolerance is lower bounded by T) = To(n, size(f)), whose query space 
and hypothesis class are Qo and ‘Ho, respectively, and whose output hypothesis has error rate at 
most $37, then a strong SQ learning algorithm can be constructed whose query complexity is 
O(N log* +) and whose tolerance is Q(t )¢y/log(1/¢)). The query space complexity is given 


by 
log |Qz| = O(log |Qo| + sz log = log |Ho]) 
when Qy and Hoy are finite, or 
VC(Qp) = O( VC(Qo) + VC(Ho) - (<3 log Z) log( <5 log =) 


when Q, and Ho have finite VC-dimension. 


3.1.3. Hybrid Boosting in the Statistical Query Model 


We obtain a more efficient boosting scheme in the SQ model by combining the two previously 


described methods. As in the PAC model, we use Scheme 1 to boost from $7 to ; and 
Scheme 2 to boost from 7 to «. By combining the results of Theorem 1 and Theorem 2, we 
immediately obtain an upper bound on the query complexity of the hybrid boosting scheme 
and a lower bound on the tolerance of the hybrid boosting scheme. An upper bound on the 
query space complexity of the hybrid boosting scheme is given in the appendix. We thus obtain 


the following improved boosting result. 


Theorem 3 Given a weak SQ learning algorithm whose query complexity is upper bounded by 
No = No(n, size(f)), whose tolerance is lower bounded by T) = To(n, size(f)), whose query space 
and hypothesis class are Qo and ‘Ho, respectively, and whose output hypothesis has error rate at 
most $27, then a strong SQ learning algorithm can be constructed whose query complexity is 


O(N log* +) and whose tolerance is Q(T ¢/log(1/¢)). The query space complexity is given by 
log |Que| = O(log | Qo] + =z log = log | Hol) 
when Qy and Hoy are finite, or 


VC(Qup) = O(VC(Qo) + VC(Ho) « (4 10g 4) log( +s log 4)) 
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when Q, and Ho have finite VC-dimension. 


Note that the tolerance of the strong SQ learning algorithm constructed has no dependence 


on ¥ in this hybrid boosting scheme. 


3.2 General Bounds on Learning in the Statistical Query Model 


In this section, we derive general upper bounds on the complexity of statistical query learning. 
These results are obtained by applying the boosting results of the previous section. We further 
show that our general upper bounds are nearly optimal by demonstrating the existence of a 


function class whose minimum learning complexity nearly matches our general upper bounds. 


3.2.1 General Upper Bounds on Learning in the SQ Model 


Just as the sample complexity of boosting in the PAC model yields general upper bounds on the 
sample complexity of strong PAC learning, the query, query space and tolerance complexities 
of boosting in the SQ model yield general bounds on the query, query space and tolerance 
complexities of strong 5Q learning. 

We can convert any strong SQ learning algorithm into a weak SQ learning algorithm by 
“hardwiring” the accuracy parameter € to a constant. We can then boost this learning algo- 
rithm, via Scheme 2 for instance, to obtain a strong SQ learning algorithm whose dependence 


on € is nearly optimal. 


Theorem 4 /f the class F is strongly SQ learnable, then F is strongly SQ learnable by an al- 
gorithm whose query complexity is O(No log” +), whose tolerance is Q(T9¢/ log(1/¢€)), and whose 
query space complexity is O(p3(n) log +) when the query space is finite or O(p4(n) log + log log +) 
when the query space has finite VC-dimension, where No = pi(n, size(f)), T = 1/po(n, size(f)) 


and p,, Pz, p3 and p4, are polynomials. 


While we have focused primarily on the query, query space and tolerance complexities of 
5Q learning, we note that our boosting results can also be applied to bound the time, space 
and hypothesis size complexities of SQ learning. It is easily shown that, with respect to €, these 


complexities are bounded by O(log” +), O(log +) and O(log +), respectively. 
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For any function class of VC-dimension d, Kearns [19] has shown that learning in the SQ 
model requires Q(d/logd) queries each with additive error O(c). Whereas Kearns simultane- 
ously lower bounds the query complexity and upper bounds the tolerance, we have simultane- 
ously upper bounded the query complexity and lower bounded the tolerance. Note that the 
tolerance we give in Theorem 4 is optimal to within a logarithmic factor. While Kearns’ gen- 
eral lower bound leaves open the possibility that there may exist a general upper bound on the 


query complexity which is independent of €, we show that this is not the case by demonstrating 


a 


a specific learning problem which requires Os 


log +) queries each with additive error O(¢) in 
the SQ model. Thus, with respect to €, our general upper bound on query complexity is within 


a log + factor of the best possible general upper bound. 


3.2.2 A Specific Lower Bound for Learning in the SQ Model 


In this section, we describe a function class whose minimum learning complexity nearly matches 
our general upper bounds. We begin by introducing a game on which our learning problem is 
based. 

Consider the following two player game parameterized by t, d and N wheret <d< N. The 
adversary chooses a set” S' C [N] of size d, and the goal of the player is to output a set T C [N] 
such that |S AT| < ¢. The player is allowed to ask queries of the form @ C [N] to which the 


adversary returns |Q 9 S|. 


Lemma 1 For any d > 4, t < d/4 and N = Q(d'*?) for some a > 0, the player requires 


Uga log N) queries of the oracle, in the worst case. 


Proof: Any legal adversary must return responses to the given queries which are consistent 
with some set $ C [N] of size d. We construct an adaptive, malicious adversary which works 
as follows. Let Sy C 2l! be the set of all (*) subsets of size d. When the player presents 
the first query Q, C [N], the adversary calculates the value of |.5. Q,| for every S € Sy and 
partitions the set Sp into d+ 1 sets S$,Sj,...,S% where each subset S € Si has [591 Q,| = 7. 
For i = arg max, {|S3|}, the adversary returns the value i and lets S, = Sj. In general, S, is 


the set of remaining subsets which are consistent with the responses given to the first & queries, 


?We use the standard combinatorial notation [NV] = {1,...,.N}. 
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and the adversary answers each query so as to maximize the remaining number of subsets. Note 
that |S4| > |Sol/(d + 1) = () (d+ 1). 

For any S C 2!], we define width(S) = maxs,s,es {|5; A S;|}. Note that if width(S;,) > 2t, 
then there exist at least two sets $1, 5. € S, such that |S; A S,| > 2t. This implies that there 
cannot exist a set T’ which satisfies both |.$, AT| < ¢ and |S, AT| < t (since A is a metric over 
the space of sets which satisfies the triangle inequality property). If the player were to stop 
and output a set 7 at this point, then the malicious adversary could always force the player to 
lose. We now bound width(S,) as a function of |S,|. This, combined with our bound on |S;| as 
a function of k, will effectively bound the minimum number of queries required by the player. 


We make use of the following inequalities:? 


n\" n n en\" 
GC) Oss @) 
r r r r 
For any S C 20! of width at most w, one can easily show that |S| < (*). Thus, if 


|S,| > (2%), then width(S,) > 2t. We now note that any k which satisfies the following 


2t 


inequality will guarantee that width(S;,) > 2t: 


(2) < Cae) s (=) < Ae < a0 < |S 


Solving the third inequality for (d + 1)*, we obtain: 


wear <(G) CR) = (za) 


Thus, a lower bound on the number of queries required by the player is 


d N 
2 log 2ed 


d 
2 8 2d = (“log N 
log(d + I) (ma 06.) 


for N = Q(d'**), 


Now consider a learning problem defined as follows. Our instance space X is the set of 


natural numbers NV, and our function class is the set of all indicator functions corresponding to 


°We use the standard combinatorial notation ((")) = io (") . 
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subsets of NV’ of size d. This function class is easily learnable in the SQ model. In what follows, 


4 


we show that any deterministic 5Q algorithm for this class requires Ona 


log +) queries with 


additive error O(€). 


Theorem 5 There exists a parameterized family of function classes which requires Una log +) 


queries with additive error O(€) to learn in the SQ model. 


Proof: Consider the two-player game as defined above. For an instance of the game specified 
by t, d and N (where d > 4,t = d/4 and N = Q(d'*°)), we create an instance of the learning 
problem as follows. We define our distribution D over N to place weight 4/Nd on each point 
1,..., N and to place weight 1<4/d on the point N+1. All other points have zero weight. We 
set € = 1/N and call the deterministic SQ learning algorithm. Note that the target subset has 
weight 4/N, so if the SQ algorithm submits a query with additive error greater than 4¢ = 4/N 
we may answer the query ourselves (as if the target subset were “empty”). For any query x 
submitted with tolerance less than 4¢, we determine the exact answer as follows. Begin with 
an answer of 0. If x(N + 1,0) = 1, then add 1 <4/d to the answer. Determine the following 
three subsets of [N]: X?, Xj and X, where x € XP if y(z,0) = 1 and y(2,1) = 0, 2 € X} if 
x(#,0) = 0 and y(a,1) = 1, and « € Xy if x(#,0) = 1 and x(a,1) = 1. Add |X2|-4/Nd to the 
answer. Submit the query X? to the adversary, and for a response r add (|X7| @r)-4/Nd to 
the answer. Submit the query Xj to the adversary, and for a response r add r-4/Nd to the 
answer. Return the final value of the answer to the 5Q algorithm. 

Note that we are able to answer each SQ algorithm query by submitting only two queries 


to the adversary, and we need not submit any queries to the adversary if the requested additive 


d 
log d 


error is greater than 4e. Since Q( log N) queries of the adversary are required, the SQ 


algorithm must ask Una log N) = Una log +) queries with additive error O(c). 


Using techniques similar to those found in Kearns’ lower bound proof [19], the above proof 
can easily be modified to show that even if the adversary chooses his subset randomly and 


uniformly before the game starts, then there exists some constant probability with which any 


dlog(1/e) 


Ioatdlogti/o) ) queries with 


SQ algorithm (deterministic or probabilistic) will fail if it asks o( 


additive error O(¢). This result is given in the appendix. 
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3.3 Simulating SQ Algorithms in the Classification Noise Model 


In this section, we describe an improved method for efficiently simulating a statistical query 
algorithm in the classification noise model. The advantages of this new method are twofold. 
First, our simulation employs a new technique which significantly reduces the running time of 
simulating SQ algorithms. Second, our formulation for estimating individual queries is simpler 
and more easily generalized. 

Kearns’ procedure for simulating 5Q algorithms works in the following way. Kearns shows 
that given a query y, P, can be written as an expression involving the unknown noise rate 7 
and other probabilities which can be estimated from the noisy example oracle FX7,,(f, D). We 
note that the derivation of this expression relies on y being {0,1}-valued. The actual expression 


obtained is given below. 


ol 
~ 162n 


x 


i UI 
Pp! 1<o——_ P? o—_ 3.7 
yor ( *Tay) Doky Tea, (3.7) 


In order to estimate P, with additive error 7, a sensitivity analysis is employed to determine 
how accurately each of the components on the right-hand side of Equation 3.7 must be known. 
Kearns shows that for some constants c, and C2, if 7 is estimated with additive error c;7(1 427)? 
and each of the probabilities is estimated with additive error co7(1 <@2y,), then the estimate 
obtained for P, from Equation 3.7 will be sufficiently accurate. Since the value of 7 is not 
known, the procedure for simulating SQ algorithms essentially guesses a set of values for 7, 
{no,m,---,m}, such that at least one 1; satisfies |n; 7| < cit.(1 27)? where 7, is a lower 
bound on the tolerance of the SQ algorithm. Since c,7,(142m)? < c17.(1 27)’, the simulation 


uniformly guesses O( ) values of 7 between 0 and 7. For each guess of 7, the simulation 


l 
Ta(1— 270)? 
runs a separate copy of the SQ algorithm and estimates the various queries using the formula 
given above. Since some guess at 7 was good, at least one of the runs will have produced a 
good hypothesis with high probability. The various hypotheses are then tested to find a good 
hypothesis, of which at least one exists. Note that the 7-guessing has a significant impact on 
the running time of the simulation. 

In what follows, we show a new derivation of P, which is simpler and more easily gener- 
it is 


alizable than Kearns’ original version. We also show that to estimate an individual P,, 
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only necessary to have an estimate of 7 within additive error cr(1 27) for some constant c. 
We further show that the number of 7-guesses need only be O(+ log im) thus significantly 


reducing the time complexity of the SQ simulation. 


3.3.1 A New Derivation for P, 


In this section, we present a simpler derivation of an expression for Py. In previous sections, 
it was convenient to view a {0,1}-valued y as a predicate so that P, = Prp[x(2, f(#))]. In 
this section, it will be more convenient to view y as a function so that P, = Ep[x(2, f(2))]. 
Further, by making no assumptions on the range of y, the results obtained herein can easily be 
generalized; these generalizations are discussed in Chapter 5. 

Let X be the instance space, and let Y = X x {0,1} be the labelled example space. We 
consider a number of different examples oracles and the distributions these example oracles 
impose on the space of labelled examples. For a given target function f and distribution D 
over X, let KX(f,D) be the standard, noise-free example oracle. In addition, we define the 
following example oracles: Let EX(f,D) be the anti-example oracle, EX'.,(f,D) be the noisy 
example oracle and EX?.,.(f,D) be the noisy anti-example oracle. Note that we have access 
to EX?.,(f,D) and we can easily construct EX?.,(f,D) by simply flipping the label of each 
example drawn from hX2,(f, D). 

Each of these oracles imposes a distribution over labelled examples. Let D;, D;, dD’ and 
Di be these distributions, respectively. Note that P, = Ep[x(, f(«))] = Ep, [x]. 

Finally, for a labelled example y = (2,1), let 7 = (2,1). We define Y(y) = (7%). Note that 
Y is a new function which, on input (x,/), simply outputs y(a,/). The function Y is easily 


constructed from y. 


Theorem 6 
_ (1 0)Eps[x] @7Ep: IX] 


P. =Ep,[x] = oF (3.8) 


Proof: We begin by relating the various example oracles defined above. Recall that the noisy 
example oracle FX?2,(f, D) is defined as follows: Draw an instance « € X according to D 
and output (2, f(x)) with probability 1 <n or (a,7f(a)) with probability 7. The draw of « 


is performed randomly and independently for each call to FX2,(f,D), and the correct or 
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incorrect labelling of a is performed randomly and independently for each call to EX2,(f, D). 
In particular, the correct or incorrect labelling of # is not dependent on the instance 2 itself. 

Given the independence described above, we may equivalently define EX2,(f,D) (and 
EX"..(f, D)) as follows: 


EX"..(f, D) EX(f,D) with probability 1 


EX(f,D) with probability n 


EX",(f,D) = EX(f,D) with probability 1 


EX(f,D) with probability n 


We may use these equivalent definitions to deduce the following: 


Ep:|x] = (1)Ep,[x] + 7Ep,[y] (3.9) 


Ep:|[x]_ = (1 7)Ep,[x] + 7Ep, [x] (3.10) 


Multiplying Equation 3.9 by (1 47) and Equation 3.10 by 7, we obtain: 


(L@n)Ep:[x] = (1 )*Ep, [x] +970 )Eb,[y] (3.11) 


nEp[x] = (1 n)Ep,[x] + 1°Ep, [yx] (3.12) 


Subtracting Equation 3.12 from Equation 3.11 and solving for Ep,[x], we finally obtain: 


(1 7)Ep:[x] @nEp7[x] 


Ep, [x] = 162 


To obtain Equation 3.8, we simply note that Ep? [x] = Ep: [X]- 
Note that in the derivation given above, we have not assumed that x is {0,1}-valued. This 
derivation is quite general and can be applied to estimating the expectations of real-valued 


queries. This result is given in Chapter 5. 
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Finally, note that if we define 


y"(y) = (1 Sen) 


then P, = Ep,[x] = Ep: x"). Thus, given a y whose expectation we require with respect to 
the noise-free oracle, we can construct a new y whose expectation with respect to the noisy 
oracle is identical to the answer we require. This formulation may even be more convenient if 
one has the capability of estimating the expectation of real-valued functions; we discuss this 


generalization in Chapter 5. 


3.3.2 Sensitivity Analysis 


In this section, we provide a sensitivity analysis of Equation 3.8 in order to determine the 


accuracy with which various quantities must be estimated. We make use of the following claim. 


Claim 3 /f 0<a,b,c,7 <1 and {a = b/c, a= b-c, a=bsSc}, then to obtain an estimate of 
a within additive error T, it is sufficient to obtain estimates of 6 and c within additive error 


{er /3, T(V/2 $1), 7/2}, respectively. 


Proof: The a = b/c case is proven in Claim 1. The a = 6-c¢ case is proven as follows. 


(b+7(V281))-(e+ 7(V2 61)) = b-e+br(V2 61) 4 er(V2 61) 4 7(V2 61) 
= atbr(V2 41) +er(V2 $1) + 7°(3 62V2) 
< a47(V261)4+7(V2 61) 4+7(3 62v2) 


(b S7(V2 $1)) -(e 67(V2 $1)) b-¢ Sbr(V2 61) Ser(V2 $1) +. 7°(V2 61) 
= abr(V2 $1) Ser(V2 $1) + 7°73 62Vv2) 
ao7(V261) 67(vV2 41) 


aDT 


IV 


IV 


The a = b Se case is trivial. 
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Lemma 2 Let 4), Ep: [x] and Ep: [X] be estimates of n, Ep: [x] and Ep: [x] each within additive 
error T(12n)(V2 $1)/6. Then the quantity 


(1&9) Ep: [x] Ep» [Tl] 
1627 


is within additive error rT of Py, = Ep,[x]. 


Proof: To obtain an estimate of the right-hand side of Equation 3.8 within additive error 7, 
it is sufficient to obtain estimates of the numerator and denominator within additive error 
(1 $2n)7/3. This condition holds for the denominator if 7 is estimated with additive error 
(1 2n)7r/6. 

To obtain an estimate of the numerator within additive error (1 <@27)7/3, it is sufficient to 
estimate the summands of the numerator with additive error (1 <27)7/6. Similarly, to obtain 


accurate estimates of these summands, it is sufficient to estimate 7, Ep» [x] and Ep: [XY] each 


with additive error (1 427)r(V/2 $1)/6. 


Estimates for Ep: |x] and Ep: [Y] are obtained by sampling, and an “estimate” for 7 is 


obtained by guessing. We address these issues in the following sections. 


3.3.3. Estimating Ep»[x] and Ep»[X] 


One can estimate the expected values of all queries submitted by drawing separate samples 
for each of the corresponding x and Y’s and applying Lemma 2. However, better results are 
obtained by appealing to uniform convergence. 

Let Q be the query space of the SQ algorithm and let O = {y: y € Q}. The query space of 
our simulation is Q’ = QU Q. Note that for finite Q, |Q’| < 2/Q|. One can further show that 
for all QO, VC(Q’) < c- VC(Q) for a constant c » 4.66. This result is given in the appendix. 

If 7, is a lower bound on the minimum additive error requested by the SQ algorithm and 
m is an upper bound on the noise rate, then by Lemma 2, (1 42m)r.(\/2 $1)/6 is a sufficient 
additive error with which to estimate all expectations. Standard uniform convergence results 


can be applied to show that all expectations can be estimated within the given additive error 
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using a single noisy sample of size 


7 1 2) 
m= (aq <> 2m )? log 6 


in the case of a finite query space, or a single noisy sample of size 


7 VC(Q) 1 1 1 
m= 0 (=n S21 )° 108 T.(1 2m) r T2(1 42m)? ) 


in the case of an infinite query space of finite VC-dimension. 


3.3.4 Guessing the Noise Rate 7 


By Lemma 2, to obtain an estimate for P,, it is sufficient to have an estimate of the noise rate 
n within additive error (1 427)r.(/2 41)/6. Since the noise rate is unknown, the simulation 
guesses various values of the noise rate and runs the SQ algorithm for each guess. If one of the 
noise rate guesses is sufficiently accurate, then the corresponding run of the $Q algorithm will 
produce the desired accurate hypothesis. 


To guarantee that an accurate 7-guess is used, one could simply guess O( values 


a=) 


of 7 spaced uniformly between 0 and 7. This is essentially the approach adopted by Kearns. 


Note that this would cause the simulation to run the 5Q algorithm O( ) times. 


a 
T+(1—2n2) 


1 
1-2 


We now show that this “branching factor” can be reduced to O(+ log ) by constructing 


our 7-guesses in a much better way. The result follows immediately from the following lemma 


when 7 = (V2 41)/6 


Lemma 3 For all y, < 1/2, there exists a sequence of n-guesses {,m,..-,m%} where i = 


O(= log Tn) such that for ally € [0,m], there exists an n; which satisfies |n<nj| < y(1 27). 


1-2 


Proof: The sequence is constructed as follows. Let 7 = 0 and consider how to determine 7; 
from j;-1. The value 7;_; is a valid estimate for all 7 > n;-1 which satisfy 7 @7(121) < nj-1. 


Consider an 


Bima t 1), 


Solving for 7, we find that ;_, is a valid estimate for all 7 € [nj-1, a 


nj-1 ty 
1427 ° 


nj; > The value 7; is a valid estimate for all 7 < ; which satisfy 7 + y(1 #27) > nj. 


Solving for 7, we find that 7; is a valid estimate for all 7 € [iS nj]. To ensure that either 7;_1 
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or 1; is a valid estimate for any 7 € [7j;-1,;], we set 


Mj-1t 7 _ My 7 
1+27 1827 


Solving for 7; in terms of n;_,, we obtain 


— ley 27 


Substituting 7’ = 2y/(1+ 27), we obtain the following recurrence: 


ny = (1 29’) ja +7’ 


Note that if y < 1/2, then 7’ < 1/2 as well. 
By constructing -guesses using this recurrence, we ensure that for all 7 € [0,7], at least 


one of {7,...,7;} is a valid estimate. Solving this recurrence, we find that 
me = 7 DOC 2y'F + mod 29/7. 


Since 7. = 0 and we are only concerned with 7 < ™, we may bound the number of guesses 


required by finding the smallest i which satisfies 


t—1 


7 DOU 29'" > 


j=o 


Given that 
i-1 


7 OL 27')) = 7 


j=o 


1o(1427'))  1e(1 627’)! 
1e(1 627’) — 2 


we need (1 42y')' < 12m. Solving for 7, we find that any 7 > In +—/In 


1_ is sufficient. 
1-2np Y 


1-2 


Using the fact that 1/a > 1/1In + for all « € (0,1), we find that 


1 1 1+ 2y 1 
t= —-ln = -In 
2y' 1S2m Ay 1S2m 


is an upper bound on the number of guesses required. 
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3.3.5 The Overall Simulation 


We now combine the results of the previous sections to obtain an overall simulation as follows: 


1. Draw m, labelled examples from EX%..(f,D) in order to estimate the expectations in 


Step 2. 


2. Run the SQ algorithm once for each of the O(S log icmp) n-guesses, estimating the various 


queries by applying Lemma 2 and using the sample drawn. 


3. Draw mz, samples and test the o(+ log cm) hypotheses obtained in Step 2. Output one 


of these hypotheses whose error rate is at most e. 


Step 3 can be accomplished by a generalization of a technique due to Laird [21]. The sample 


size required is 


1 1 1 
= ————~ log(— | . 
m=O (a <> 2m )? og OT, °8 1 a)) 
Since 1/7, = Q(1/e) for all SQ algorithms [19], we obtain the following theorem on the total 


sample complexity of this simulation. 


Theorem 7 If F is learnable by a statistical query algorithm which makes queries from query 
space Q with worst case additive error 7,, then F is PAC learnable in the presence of classifica- 
tion noise. If n, < 1/2 is an upper bound on the noise rate, then the sample complexity required 
18 
[el 
ol; Timms los + sama los log im) 


when Q is finite or 


O( aac bes sate + = blog +) 


when Q has finite VC-dimension. 


By combining our results on general bounds for 5Q learning and classification noise simu- 


lation, we immediately obtain the following corollary. 


Corollary 1 If F is SQ learnable, then F is PAC learnable in the presence of classification 


noise. The dependence on € and m of the required sample complexity is Oana): 
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To determine the running time of our simulation, one must distinguish between two different 
types of SQ algorithms. Some 5Q algorithms submit a fixed set of queries independent of 
the estimates they receive for previous queries. We refer to these algorithms as “batch” SQ 
algorithms. Other $Q algorithms submit various queries based upon the estimates they receive 
for previous queries. We refer to these algorithms as “dynamic” SQ algorithms.* Note that 
multiple runs of a dynamic $Q algorithm may produce many more queries which need to 
be estimated. Since the vast majority of the time required to simulate most 5Q algorithms 
is spent estimating queries using a large sample, the time complexity of simulating dynamic 
5Q algorithms is greatly affected by the “branching factor” of the simulation. By reducing 
the “branching factor” of the simulation from O=oa) to O(+ log ica) the asymptotic 
running time of our simulation is greatly improved. 


With respect to 7, the running time of our simulation is O( . Simon [30] has shown 


Tam) 


a sample and time complexity lower bound of Q( for PAC learning in the presence of 


ce) 


classification noise. We therefore note that the running time of our simulation is optimal with 
respect to the noise rate (modulo lower order logarithmic factors). For dynamic algorithms, 


the time complexity of our new simulation is in fact a O( factor better than the current 


ry ) 


simulation. 


“Note that we consider any SQ algorithm which uses a polynomially sized query space to be a “batch” 
algorithm since all queries may be processed in advance. 


CHAPTER 4 


Learning Results in the 
Relative Error SQ Model 


In this chapter, we propose a new model of statistical query learning based on relative error. 
We show that learnability in this new model is polynomially equivalent to learnability in the 
standard, additive error model; however, this new model is advantageous in that 5Q algorithms 


specified in this model can be simulated more efficiently in some important cases. 


4.1 Introduction 


In the standard model of statistical query learning, a learning algorithm asks for an estimate 
of the probability that a predicate y is true. The required accuracy of this estimate is specified 
by the learner in the form of an additive error parameter. The limitation of this model is 
clearly evident in even the standard, noise-free statistical query simulation [19]. This simula- 
tion uses 2(1/72) examples. Since 1/7, = 0(1/e) for all SQ algorithms [19], this simulation 
effectively uses Q(1/e?) examples. However, the «dependence of the general bound on the 
sample complexity of PAC learning is O(1/e) [7, 11]. 

This Q(1/72) = Q(1/e*) sample complexity results from the worst case assumption that 
large probabilities may need to be estimated with small additive error in the SQ model. Either 
the nature of statistical query learning is such that learning sometimes requires the estimation of 


large probabilities with small additive error, or it is always sufficient to estimate each probability 
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with an additive error comparable to the probability. If the former were the case, then the 
present model and simulations would be the best that one could hope for. We show that the 
latter is true, and that a model in which queries are specified with relative error is a more 
natural and strictly more powerful tool. 

We define such a model of relative error statistical query learning and we show how this 
new model relates to the standard additive error model. We also show general upper bounds 
on learning in this new model which demonstrate that for all classes learnable by statistical 
queries, it is sufficient to make estimates with relative error independent of «. We then give 
roughly optimal PAC simulations for relative error SQ algorithms. Finally, we demonstrate 


natural problems which only require estimates with constant relative error. 


4.2 The Relative Error Statistical Query Model 


Given the motivation above, we modify the standard model of statistical query learning to allow 
for estimates being requested with relative error. We replace the additive error STAT(f, D) 
oracle with a relative error Rel-STAT(f,D) oracle which accepts a query x, a relative error 
parameter y, and a threshold parameter 6. The value P, = Prp[x(2, f(x))] is defined as before. 
If Py is less than the threshold @, then the oracle may return the symbol 1. If the oracle does 


not return L, then it must return an estimate Py such that 
Pep) <P <P+p) 


Note that the oracle may chose to return an accurate estimate even if P, < @. A class is said 
to be learnable by relative error statistical queries if it satisfies the same conditions of additive 
error statistical query learning except we instead require that 1/f and 1/6 are polynomially 
bounded. Let p. and 6, be the lower bounds on the relative error and threshold of every query 
made by an SQ algorithm. Given this definition of relative error statistical query learning, we 


show the following desirable equivalence. 


Theorem 8 F is learnable by additive error statistical queries if and only if F is learnable by 


relative error statistical queries. 
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Proof: One can take any query y to the additive error oracle which requires additive error 7 
and simulate it by calling the relative error oracle with relative error 7 and threshold 7. If 
Py = L, then return 0; else, return Py. 

Similarly, one can take any query to the relative error oracle which requires relative error js 
and threshold @ and simulate it by calling the additive error oracle with additive error 6/3. If 
Py < 6(1 Sy/3), then return L; else, return Py. 


In each direction, the simulation uses polynomially bounded parameters if and only if the 


original algorithm uses polynomially bounded parameters. 


Kearns [19] shows that almost all classes known to be PAC learnable are learnable with 
additive error statistical queries. By the above theorem, these classes are also learnable with 
relative error statistical queries. In addition, the hardness results of Kearns [19] for learning 
parity functions and the general hardness results of Blum e¢ al. [6] based on Fourier analysis 


also hold for relative error statistical query learning. 


4.3. A Natural Example of Relative Error SQ Learning 


In this section we examine a learning problem which has both a simple additive error SQ 
algorithm and a simple relative error SQ algorithm. We consider the problem of learning a 
monotone conjunction of Boolean variables in which the learning algorithm must determine 
which subset of the variables {2,,...,2,} are contained in the unknown target conjunction f. 

We construct an hypothesis h which contains all the variables in the target function f, and 
thus A will not misclassify any negative examples. We further guarantee that for each variable 
a; in h, the distribution weight of examples which satisfy “«; = 0 and f(x) = 1” is at most e/n. 
Therefore, the distribution weight of positive examples which A will misclassify is at most e. 
Such an hypothesis has error rate at most €. 


Consider the following query: y;(#,!) = [(#; =0)A (l= 1)]. P. 


x, is simply the probability 
that x; is false and f(z) is true. If variable 2; is in f, then P,, = 0. If we mistakenly include a 
variable x; in our hypothesis which is not in f, then the error due to this inclusion is at most 
P,,. We simply construct our hypothesis by including all target variables, but no variables 2; 


for which Py, > €/n. 
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An additive error SQ algorithm queries each y; with additive error ¢/2n and includes all 
variables for which the estimate Py, < ¢€/2n. Even if P,, = 1/2, the oracle is constrained to 
return an estimate with additive error less than €/2n. A relative error SQ algorithm queries each 
x; with relative error 1/2 and threshold ¢/n and includes all variables for which the estimate 
Py, is 0 or L. 

The sample complexity of the standard, noise-free PAC simulation of additive error 5Q algo- 
rithms depends linearly on 1/72 [19], while in Section 4.5, we show that the sample complexity 
of a noise-free PAC simulation of relative error SQ algorithms depends linearly on 1/,:20,. Note 
that in the above algorithms for learning conjunctions, 1/72 = O(n?/e?) while 1/120, = O(n/e). 
We further note that wu, is constant for learning conjunctions. We show in Section 4.4 that no 
learning problem requires y, to depend on ¢ and in Section 4.6 that yu, is actually a constant 


in many algorithms. 


4.4 General Bounds on Learning in the Relative Error SQ Model 


In this section, we prove general upper bounds on the complexity of relative error statistical 
query learning. We do so by applying boosting techniques [14, 15, 28] and specifically, these 
techniques as applied in the statistical query model. We first prove some useful lemmas which 


allow us to decompose relative error estimates of ratios and sums. 


Lemma 4 /f 0 < a,b,c, u,0,® < 1 anda = b/c, then to estimate a with (u,@) error provided 
that c > ®, it is sufficient to estimate c with (u/3,®) error and b with (j/3,0®/2) error. 


Proof: If the estimate é¢ is 1 or less than (1 y/3), then ¢ < ©. Therefore an estimate for a 


is not required, and we may halt. Otherwise ¢ > 6(1 Sj/3), and therefore c > oil > 6/2. 


If the estimate b is 1, then b < 60/2. Therefore a = b/c < 0, so we may answer @ = L. 


Otherwise, b and é are estimates of 6 and c, each within a 1 + 4/3 factor. The theorem follows 


by noting the following facts. 


4.4 
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b-(1+p/3) 
c+ (1 p/3) 
< 
< 
b-(1n/3) 
e-(1+ 2/3) 
> 
> 


1+ 4/3 
1ep/3 
a-(1+ p/3) (1+ 5) 


(1+ 2/3 + 4/2 + p°/6) 


( 

a+ (1+ 2/3) + 4/2) 
( 

“(1+ 4) 


(1 Sp1/3) (1 ott.) 


1 op/3) @u/3) 


-(162u/3 + 2/9) 


rr a a 


(Ley) 


Lemma 5 /f 0 < s,p;,2:,u < 1, ;p; < 1 and s = ¥°,p;2;, then to estimate s with (u, 0) 


error, it is sufficient to estimate each z; with (41/3, 40/3) error provided that the p; coefficients 


are known. 


Proof: Let B = {i: estimate of z; is L}, EF = {i: estimate of z; is 2;}, s, = 3°, piz% and 


Sp = >-,pizi. Note that 5, < 64/3. Let §, = >>, p:2;. If 8, < 6(1 Sp/3)? then we return L, 


otherwise we return §,. 


If 8, < A(1ep/3)’, then s, < @(1<yu/3). But in this case s = s,+s8, < 0(1ap/3)+6u/3 = 9, 


so we are correct 


in returning L. 


Otherwise we return §, which is at least 0(1 @y/3)’. If B = 0, then it is easy to see that 


8, is within a 1 4 


t jt/3 (and therefore 1 4 


t 4) factor of s. Otherwise, we are implicitly setting 


2 = 0 for each 7 € B, and therefore it is enough to show that §, > s(1 yp). 
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Since &, > O(1 @p/3)°, we have s, > 0(1 Sy/3)?/(1 + «/3). Using the fact that for all 


w<1, (1 Sp /3)/(1+ w/3) > 1/2, we have s, > O11 Sy/3)/2. If 8, > (04/3 + s5)(1 Sy), then 


8, > s(1 Sp) since s, < Ou/3 and s = s, + sz. But since &j > 5,(1 y/3), this condition 
holds when s,(1 6/3) > (04/3 + s,)(1 Sp). Solving for sz, this final condition holds when 


Sy > O(1 Sp)/2 which we have shown to be true whenever an estimate is returned. 


Theorem 9 If the concept class F is strongly SQ learnable, then F is strongly SQ learnable by 
an algorithm whose query complexity is O( No log? +), whose minimum requested relative error is 
Q(pi9) and whose minimum requested threshold is Q(pio9€/ log(1/e)) where No = pi(n, size(f)), 


Ho = 1/po(n, size( f)) and 0) = 1/ps(n, size(f)) for some polynomials p,, py and ps. 


Proof: If F is strongly SQ learnable, then there exists a relative error statistical query algo- 
rithm A for learning #. Hardwire the accuracy parameter of A to 1/4 and apply Scheme 2 
boosting. The boosting scheme will run 16 In(1/e) copies of A with respect to 16 In(1/e) different 
distributions over the instance space. Each run makes at most No = N,(1/4,n, size(f)) queries, 
each with relative error no smaller than fig = w.(1/4, 7, size( f)) and threshold no smaller than 
6 = 0,(1/4, n, size( f)). In run i+1, the algorithm makes queries to STAT(f, Dj41) where Dj41 
is a distribution based on D. Since we only have access to a statistics oracle for D, queries to 


STAT(f, Dizi) are simulated by a sequence of new queries to STAT(f, D) as follows: 


iso Ay STATS, DIX A x7] 


j=0 “j 


dijo Aj STAT(F, D)[X7'] 


J=0 7 


STAT(S, Diya)IX(@, S(2))] = (4.1) 


In the above equation w < i, the values A? € [0,1] are known, and 37, A¥ < 1. Also note 


e(1—e) 


Ain(1/e)? then the query need not be 


that if the denominator of Equation 4.1 is less than ® = 
estimated (this is the “abort” condition of Scheme 2). Applying Lemmas 4 and 5, we find that 
the queries in the denominator can be estimated with (10/9, o®/9) error, and the queries in 


the numerator can be estimated with (,1o0/9, 090/18) error. Since a query to STAT(f, Di+1) 


requires O(i) queries to STAT(f, D), the total number of queries made is O( Np log*(1/e)). 


We finally note that the query space complexity obtained here is identical in form to the 


query space complexity obtained in Section 3.1.2. 
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4.5 Simulating Relative Error SQ Algorithms in the PAC Model 


In this section, we derive the complexity of simulating relative error SQ algorithms in the PAC 
model, both in the absence and presence of noise. We also give general upper bounds on the 
complexity of PAC algorithms derived from 5Q algorithms based on the simulations and the 
general bounds of the previous section. Note that there do not exist two-sided bounds for 
uniform convergence based on VC-dimension, so some of our results are based on drawing a 


separate sample for each query. 


4.5.1 PAC Model Simulation 


The simulation of relative error SQ algorithms in the noise-free PAC model is based on a 
Chernoff bound analysis. Let GE(p,m,n) be the probability of at least n successes in m 
Bernoulli trials, where each trial has probability of success p. Similarly, let LE(p,m,n) be the 
probability of at most n successes in m Bernoulli trials, where each trial has probability of 


success p. Chernoff’s bounds may then be stated as follows [3]: 


mpa? /3 


GE(p,m,mp(1+ a)) 


lA 


€ 


LE(p,m,mp(1a)) < emer? 
Furthermore, we often make use of the following properties of GF’ and LE: 


p>p => LE(p,m,n)< LE(p',m,n) (4.2) 


psp => GE(p,m,n) < GE(p',m,n) (4.3) 


We may now prove following theorem. 


Theorem 10 If F is learnable by a statistical query algorithm which makes at most N, queries 
from query space Q with worst case relative error , and worst case threshold 6,, then F is 


PAC learnable with sample complexity Oz, log lel) when Q is finite or Os log X+) when 


20 20 


drawing a separate sample for each query. 
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Proof: We first demonstrate how to estimate the value of a single query, and we then extend 
this technique to yield the desired result. Let [y,u,6] be a query to be estimated, and let 
p= P,. For a given sample of size m, let p be the fraction of examples which satisfy y. In order 
to properly estimate the value of this query, we choose m large enough to ensure that each of 


the following hold with high probabilty: 
1. If p < 6/2, then p< @. 
2. If p > 0/2, then p> 6/4. 
3. If p> 6/4, then p > (1 Sp)p. 
4. If p> 6/4, then p< (14+ p)p. 


Thus, if p < 6/2, we may output L, and if p > 0/2, we may output p. To ensure a failure 

probability of at most 6, we choose m large enough to guarantee that each of the properties 

fails to hold with probability at most 6/4. Let m = 2 In 
bb 


Suppose that p > @. Then the probability that p < 0/2 is bounded by: 


SIs 


LE(p,m,0/2) < LE(@,m,@/2) 


—mé/8 


lA 


€ 


Since m > = In 3, this probability is less than 6/4. Therefore, the probability that p > 0/2 is at 


least 16/4. Thus, we have shown the following: With probabilty at least 16/4, 
p>o = p> 6/2. 


Since Property 1 is the contrapositive of the above statement, we have shown that it will fail 
to hold with probability at most 6/4. 

Property 2 is shown to hold in a similar manner, and Properties 3 and 4 are direct conse- 
quences of Chernoff bounds. 
4|Q| 


a In ==, we can ensure that all four properties will hold for all 


Now, by choosing m = a ; 


x € Q, with probability at least 1 <6. If, on the other hand, we draw N, separate samples 
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12 
MEO 


4ANs 


each of size m = In =*, we guarantee that all four properties will hold for each of the N, 


queries estimated, with probability at least 1 <6. 


Corollary 2 If F is SQ learnable, then F is PAC learnable with a sample complexity whose 
dependence on € is @(1/e). 


Although one could use boosting techniques in the PAC model to achieve this nearly optimal 
sample complexity, these boosting techniques would result in a more complicated algorithm and 
output hypothesis (a circuit whose inputs were hypotheses from the original hypothesis class). 
If instead we have a relative error 5Q algorithm meeting the bounds of Theorem 9, then we 


achieve this PAC sample complexity directly. 


4.5.2 Classification Noise Model Simulation 


For $Q simulations in the classification noise model, we achieve the sample complexity given 
in Theorem 11 below. This sample complexity is obtained by simulating an additive error SQ 
algorithm with t = 0/3 as in Theorem 8. Although this result does not improve the sample 
complexity of 5Q simulations in the presence of classification noise, we believe that to improve 
upon this bound requires the use of relative error statistical queries for the reasons discussed 


in Section 4.1. 


Theorem 11 Jf F is learnable by a statistical query algorithm which makes queries from query 
space Q with worst case relative error pi, and worst case threshold 0,, then F is PAC learnable 
in the presence of classification noise. If m < 1/2 is an upper bound on the noise rate, then the 


sample complexity required is 


|Q| 
O (aamabaae log 8! + op log log 
when Q is finite or 


O jp log 


1 1 1 
Tea 12m)? 8 Geeta) + Team 18 5) 


when Q has finite VC-dimension. 


Corollary 3 If F is SQ learnable, then F 1s PAC learnable in the presence of classification 


noise. The dependence on € and %, of the required sample complexity is Ol ance): 
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4.5.3 Malicious Error Model Simulation 


We next consider the simulation of relative error SQ algorithms in the presence of malicious 
errors. Decatur [9] has shown that an SQ algorithm can be simulated in the presence of malicious 
errors with a maximum allowable error rate which depends on 7,, the smallest additive error 
required by the SQ algorithm. In Theorem 12, we show that an $Q algorithm can be simulated 
in the presence of malicious errors with a maximum allowable error rate and sample complexity 
which depend on p, and 6,, the minimum relative error and threshold required by the SQ 
algorithm. 

The key idea in this simulation is to draw a large enough sample such that for each query, 
the combined error in an estimate due to both the adversary and the statistical fluctuation on 
error-free examples is less than the accuracy required. We formally state this idea in the claim 


given below. 


Claim 4 Let P* be the fraction of examples satisfying x in a noise-free sample of size m, and 
let Py be the fraction of examples satisfying x in a sample of size m drawn from EX ea (fh: D). 
Then to ensure |PL oP, | <1479, it is sufficient to draw a sample of size m which simultaneously 
ensures that: 

(1) The adversary corrupts at most a7, fraction of the examples drawn from EX',.,(f,D). 
(2) [Py @P2| < te. 


Theorem 12 If F is learnable by a statistical query algorithm which makes at most N, queries 
from query space Q with worst case relative error , and worst case threshold 6,, then F is 


PAC learnable in the presence of malicious errors. The maximum allowable error rate is 3, = 


Q(1.0.), and the sample complexity required is O(> — log ll) when Q is finite or OG log *) 


205 205 


when drawing a separate sample for each query. 


Proof: We first analyze the tolerable error and sample complexity for simulating a single query 
and then determine these values for simulating the entire algorithm. 

For a given query [x, 14,4], P, is the probability with respect to the noise-free example oracle 
which needs to be estimated with (y,6) error. Assume that @ < 6/16 and let B be the actual 
fraction of the sample corrupted by the malicious adversary. We choose m large enough to 


ensure that the following hold with high probability: 
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1. If 6 < 16/16, then 8 < 10/8. 

2. If P* < 56/8, then Py < @. 

3. If Ps > 30/8, then P, > 0/4. 

4. If P, > 6/4, then P* > (1 Sp /2)Py. 
5. If PR, > 0/4, then P* < (1+ p/2)Py. 


Suppose that Properties 1 through 5 all hold. If Py < 0/2, then by Property 1, Pr< 56/8, and 
by Property 2, P, < @. Thus, we may return L. 

If, on the other hand, Py > 6/2, then by Property 1, P* > 36/8, and by Property 3, 
P, > 6/4. Property 4 then implies that P? > (1 @p/2)P,, and by Property 1, we have the 
following: 


Py 2 (Lop/2)Py 0/8 > (1 ey /2)Py uP, /2 = (1 op) Py 
By applying Property 5, we may similarly show the following: 


Pos (1+ 1/2) Py + w0/8 < (1+ 1/2) Py + oP /2 = (1+ pw) Py 


Thus, we may return Py. 

We can ensure that Properties 1 through 5 collectively hold with probability at least 136 
by letting m = —51n2. The proofs that each of these properties hold with high probability 
given this sample size are analogous to the proofs for the similar properties used in Theorem 10. 
5|Q| 


538 In ==, we can ensure that all five properties will hold for all 


Now, by choosing m = 


x € Q, with probability at least 1 <6. If, on the other hand, we draw N, separate samples 


48 In 5Nx 


rr) 77, We guarantee that all five properties will hold for each of the N, 


each of size m = 


queries estimated, with probability at least 1 <6. 


Corollary 4 If F is SQ learnable, then F is PAC learnable in the presence of malicious errors. 
The dependence on € of the maximum allowable error rate is Q(e), while the dependence on € 


of the required sample complexity is O(1/e). 


Note that we are within logarithmic factors of both the O(€) maximum allowable mali- 


cious error rate [20] and the Q(1/e) lower bound on the sample complexity of noise-free PAC 
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learning [11]. In this malicious error tolerant PAC simulation, the sample, time, space and 
hypothesis size complexities are asymptotically identical to the corresponding complexities in 


our noise-free PAC simulation. 


4.6 Very Efficient Learning in the Presence of Malicious Errors 


In previous sections, we have shown general upper bounds on the required complexity of relative 
error $Q algorithms and the efficiency of PAC algorithms derived from them. In this section, 
we describe relative error SQ algorithms which actually achieve these bounds and therefore 
have very efficient, malicious error tolerant PAC simulations. We first present a very efficient 
algorithm for learning conjunctions! in the presence of malicious errors when there are many 
irrelevant attributes. We then highlight a property of this SQ algorithm which allows for its 
efficiency, and we further show that many other SQ algorithms naturally exhibit this property 
as well. We can simulate these SQ algorithms in the malicious error model with roughly optimal 
malicious error tolerance and sample complexity. 

Decatur [9] gives an algorithm for learning conjunctions which tolerates a malicious error 
rate independent of the number of irrelevant attributes, thus depending only on the number 
of relevant attributes and the desired accuracy. This algorithm, while reasonably efficient, is 
based on an additive error SQ algorithm of Kearns [19] and therefore does not have an optimal 
sample complexity. 

We present an algorithm based on relative error statistical queries which tolerates the same 
malicious error rate and has a sample complexity whose dependence on € roughly matches the 


general lower bound for noise-free PAC learning. 


Theorem 13 The class of conjunctions of size k over n variables is PAC learnable with ma- 
licious errors. The maximum allowable malicious error rate is O( see) and the sample com- 


plexity required is O (£ log” + logn + z log+ log+) . 


Proof: We present a proof for learning monotone conjunctions of size k, and we note that this 


proof can easily be extended for learning non-monotone conjunctions of size k. 


'By duality, identical results also hold for learning disjunctions. 
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The target function f is a conjunction of & variables. We construct an hypothesis h which 
is a conjunction of r = O(klog+) variables such that the distribution weight of misclassified 
positive examples is at most €/2 and the distribution weight of misclassified negative examples 
is also at most €/2. 

First, all variables which could contribute more than ¢€/2r error on the positive examples 
are eliminated from consideration. This is accomplished by using the same queries that the 
monotone conjunction SQ algorithm of Section 4.3 uses. The queries are asked with relative 
error 1/2 and threshold €/2r. 

Next, the negative examples are greedily “covered” so that the distribution weight of mis- 
classified negative examples is no more than ¢/2. We say that a variable covers all negative 
examples for which this variable is false. We know that the set of variables in f is a cover of 
size k for the entire space of negative examples. We iteratively construct h by conjoining new 
variables such that the distribution weight of negative examples covered by each new variable 
is at least a x fraction of the distribution weight of negative examples remaining to be covered. 

Given a partially constructed hypothesis hj = 2;, Aaj, A+++ A aj,, let X; be the set of 
negative examples not covered by hy, ie. X; = {a : (f(x) = 0) A (hj(x) = 1)}. Let D; be the 
conditional distribution on X; induced by D, i.e. for any x € Xj, Dj(x) = D(x)/D(X;). By 
definition, X, is the space of negative examples and Dj is the conditional distribution on X95. 
We know that the target variables not yet in h; cover the remaining examples in X;; hence, 
there exists a cover of X; of size at most k. Thus there exists at least one variable which covers 
a set of negative examples in X; whose distribution weight with respect to D; is at least 1/k. 

Given h;, for each z;, let y;;(%,1) = [A|B] = [a; = O|(0 = 0) A (hj (xv) = 1]. Note that 
P,,,, is the distribution weight, with respect to D;, of negative examples in X; covered by 
a;. Thus there exists a variable 2; such that P,,, is at least 1/k. To find such a variable, we 
ask queries of the above form with relative error 1/3 and threshold 2/3k. [Note that this is a 
query for a conditional probability, which must be determined by the ratio of two unconditional 
probabilities. We show how to do this below.] Since there exists a variable x; such that 
Py 2 1/k, we are guaranteed to find some variable x; such that the estimate Poy is at least 
+(1 $5) = =. Note that if Poy > =, then Py, , 2 =/(1 + z) = ae Thus, by conjoining x, to 


h;, we are guaranteed to cover a set of negative examples in X; whose distribution weight with 
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respect to D; is at least 1/2k. Since the distribution weight, with respect to Dj, of uncovered 
negative examples is reduced by at least a (les) factor in each iteration, it is easy to show that 
this method requires no more than r = O(k log=) iterations to cover all but a set of negative 
examples whose distribution weight, with respect to Dj (and therefore with respect to D) is at 
most €/2. 

We now show how to estimate the conditional probability query [A|B] with relative error 
pe = 1/3 and threshold 6 = 2/3k. We estimate both queries which constitute the standard 
expansion of the conditional probability. Appealing to Lemma 4, we first estimate [B], the 
probability that a negative example is not covered by h, using relative error 4/3 = 1/9 and 
threshold ¢/2. If this estimate is L or less than £(1 4) = *, then the weight of negative 
examples misclassified by h is at most €/2, so we may halt and output h. Otherwise, we 
estimate [A A B] with relative error /3 = 1/9 and threshold 6(€/2)/2 = <. If this estimate 
is L, then we may return L, and if a value is returned, then we can return the ratio of our 


estimates for [A A B] and [B] as an estimate for [A|B]. 


For this algorithm, the worst case relative error is Q(1), the worst case threshold is Q(>57), 


and log |Q| = O(klog+logn). Therefore, the theorem follows from Theorem 12. 


An important property of this statistical query algorithm is that for every query, we need 
only to determine whether P, falls below some threshold or above some constant fraction of this 
threshold. This allows the relative error parameter yz to be a constant. The learning algorithm 
described in Section 4.3 for monotone conjunctions has this property, and we note that many 
other learning algorithms which involve “covering” also have this property (e.g. the standard 
SQ algorithms for learning decision lists and axis parallel rectangles). In all these cases we 


obtain very efficient, malicious error tolerant algorithms. 


CHAPTER 5 


Extensions 


Throughout this thesis, we have assumed that queries submitted to the statistical query oracle 
were restricted to being {0,1}-valued functions of labelled examples. In this case, the oracle 
returned an estimate of the probability that y(2, f(#)) = 1 on an example x chosen randomly 
according to D. 

We now generalize the SQ model to allow algorithms to submit queries which are real- 
valued. Formally, we define a real-valued query to be a mapping from labelled examples to the 
real interval [0,M], x : X x {0,1} — [0,M].' We define P, to be the expected value of y, 
P. = Eplx(x, f(e))] = Ev, i. 

This generalization can be quite useful. If the learning algorithm requires the expected value 
of some function of labelled examples, it may simply specify this using a real-valued query. By 
suitably constructing new queries, the learning algorithm may calculate variance and other 
moments as well. This generalization gives the algorithm designer more freedom and power. 
Furthermore, the ability to efficiently simulate these algorithms in the PAC model, in both the 
absence and presence of noise, is retained as shown below. 

The results given below are proven almost identically to their counterparts by simply ap- 
plying Hoeffding and Chernoff style bounds for bounded real random variables. The following 


is a simple extension of results contained in McDiarmid [23]: 


'The range [0, M] is used so that we can derive efficient simulations of relative error SQ algorithms. For 
additive error SQ algorithms, one may consider any interval [a,b] where M =6b—a. 
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Theorem 14 Let X,,X2,...,Xm be independent and identically distributed random variables 
where 0 < X; < M and p = E[X;j], and let p = +07", X;. For any a > 0, 


Pr[p > pt a] < e72ma?/M* 
Prip< peal < e7?mer/Me 
For anyy,0<y <1, 
Pr[p > p(l+y)) < emer eM 


Prlp < plea) < emer PM, 


Note that when M = 1, the following sample complexities and noise tolerances are essentially 


identical to those for {0, 1}-valued queries. 


Theorem 15 If F is learnable by a statistical query algorithm which makes at most N, [0,M]- 


valued queries from query space Q with worst case additive error T,, then F is PAC learnable 


with sample complexity o(4e log lel) when Q is finite or o( Ase log Xs) when drawing a sep- 


2 
Tx 


arate sample for each query. 


Theorem 16 [f F is learnable by a statistical query algorithm which makes at most N, [0, M]- 


valued queries from query space Q with worst case relative error yu, and worst case thresh- 


old 0,, then F is PAC learnable with sample complexity O(a log lel) when Q is finite or 


Oe log ==) when drawing a separate sample for each query. 


Theorem 17 If F is learnable by a statistical query algorithm which makes at most N, [0, M]- 
valued queries from query space Q with worst case additive error 7,, then F is PAC learnable 
in the presence of classification noise. If nm < 1/2 is an upper bound on the noise rate, then the 


sample complexity required is 


M? || 
O (ating los + aactag log log az) 


when Q is finite or 


N,M? Ny 
0 (=i log“ + iia log log ss) 


Extensions 69 


when drawing a separate sample for each query. 


Theorem 18 If F is learnable by a statistical query algorithm which makes at most N, [0, M]- 
valued queries from query space Q with worst case relative error pt, and worst case threshold @,, 
then F is PAC learnable in the presence of classification noise. If m < 1/2 is an upper bound 


on the noise rate, then the sample complexity required is 
[el 
O (. aS 2nv)? log ! 8 + e(1— on y? log log 3, 1- oon —.) 
when Q is finite or 
NeM 1 1 
O (. 262(1—2n,)? log “= 7 + e(1—2n, )? log log —_) 
when drawing a separate sample for each query. 


Theorem 19 I[f F is learnable by a statistical queries algorithm which makes at most N, [0, M]- 
valued queries from query space Q with worst case additive error 7,, then F is PAC learnable 


in the presence of malicious errors. The maximum allowable error rate is Q(7./M) and the 


sample complexity required is O( “> = log lel) when Q is finite or oe X=) when drawing 


a separate sample for each query. 


Theorem 20 If F is learnable by a statistical queries algorithm which makes at most N, (0, M]- 
valued queries from query space Q with worst case relative error pt, and worst case threshold 
0,, then F is PAC learnable in the presence of malicious errors. The maximum allowable error 
rate is Q(u.0,./M) and the sample complexity required is Oar x - log lel) when Q is finite or 
oe 


x) when drawing a separate sample for each query. 


CHAPTER 6 


Conclusions and Open Questions 


We have examined the statistical query model of learning and derived the first general bounds 
on the complexity of learning in this model. We have further shown that our general bounds are 
nearly optimal in many respects by demonstrating a specific class of functions whose minimum 
learning complexity nearly matches our general bounds. We have also improved the current 
strategy for simulating 5Q algorithms in the classification noise model by demonstrating a new 
simulation which is both more efficient and more easily generalized. 

The standard statistical query model of learning has a number of demonstrable deficiencies, 
and we have proposed a variant of the statistical query model based on relative error in order to 
combat these deficiencies. We have demonstrated the equivalence of additive error and relative 
error 5Q learnability, and we have derived general bounds on the complexity of learning in this 
new relative error 5Q model. We have demonstrated strategies for simulating relative error SQ 
algorithms in the PAC model, both in the absence and presence of noise. Our simulations in 
the absence of noise and in the presence of malicious errors yield nearly optimal noise-tolerant 
PAC learning algorithms. 

Finally, we have shown that our results in both the additive and relative error SQ models 
can be extended to allow for real-valued queries. 

The question of what sample complexity is required to simulate statistical query algorithms 


in the presence of classification noise remains open. The current simulations of both additive 
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and relative error $Q algorithms yield PAC algorithms whose sample complexities depend 
quadraticly on 1/e. However, in the absence of computational restrictions, all finite concept 
classes can be learned in the presence of classification noise using a sample complexity which 
depends linearly on 1/e [21]. It seems highly unlikely that a O(1/e) strategy for simulating 
additive error $Q algorithms exists; however, such a strategy for simulating relative error SQ 
algorithms seems plausible. This line of research is currently being pursued. 

As discussed in Section 4.6, many classes which are $Q learnable have algorithms with a 
constant worst case relative error u,.. Can one show that all classes which are 5Q learnable 


have algorithms with this property, or instead characterize exactly which classes do? 


CHAPTER A 


Appendix 


In this chapter, we prove a number of technical results used in the previous chapters. 


A.1 The Finite Query Space Complexity of Boosting 


In this section of the appendix, we show how to simplify the expression for the size of the query 
space of boosting and how to derive an expression for the size of the query space of hybrid 


boosting. These results apply when the query space and hypothesis class are finite. 


A.1.1. The Size of the Query Space of Boosting 


In Section 3.1, the following expression was obtained for the size of the query space of boosting: 
k . k ; 
; Ho| +2 ol ; Ho| +21 
Joni = ial ey (METS glare y (MEE!) (aay 
i=l t=1 


where & is k, or ky depending on the type of boosting used. 


We begin by simplifying the expression yoy (i+ 1)- (At), In order to obtain a closed-form 


expression for this sum, we first eliminate the (7 + 1) factor. 


vere") _ ; (N+i<1)! (“tre") 
7 (Noli! 


i+1)-( 


t t 
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_ (N+i<1)! (“A;*") 
~ NI(iel)! i 


vie ye) 
rol a 


Using the fact that 772, ("*’) = ("t™*"), we now have the following: 


k 


sity. (Ste!) vo (Ve ‘yar? i) 


t=1 t 


k-1 : k . 
N+ Nesl 
( P)+rUl . “ye 
i=0 


N+k N+k 
ve (or)+( ke jen 


Applying this fact to Equation A.1 above, we obtain the following closed-form expression: 


I 
= 
M 


7 |Ho| + *) Ge + *) 
Jn] = (lQo]-+1) (7 ") + |rtl(laol +) (Mh) et (A2) 
In order to bound the above expression, we make use of the following inequality: 
("*") — (nt m\(n+m1)---(n+1) 
m 7 mim 1)-+-1 
= (a) + rea) (+9) 
7 mol 1 
< (ltnyr 
Applying this inequality, we now have: 
[Qs] < ([Qol + 1)((Ho] + 1)* + [Hol(|Qo] + 1)(/Hol + 2)°! 1 
<_ (|Qo| + 1)([Ho] + 2)" + Qo] + 1)(Hol + 2)* 
= 2(|Qo| + 1)([Ho] + 2)" (A.3) 


The complexity of simulating an SQ algorithm depends on log|Q,|. We have effectively shown 


the following: 
log |Qz| = O(log |Qo] + k log |Ho]) 
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A.1.2. The Size of the Query Space of Hybrid Boosting 


In the hybrid boosting scheme, the Scheme 1 and Scheme 2 boosting schemes are combined to 
obtain improved overall complexities. The Scheme 2 booster uses the Scheme | booster, run 


> while the Scheme 1 booster uses the actual weak learner. 


with « = 1/4, as its “weak learner,’ 
Thus, ky = ki(y,1/4) and ky = ko(1/4,6€). Let Qyp be the query space of the hybrid booster, 
and let Q1;4 and H,;4 be the query space and hypothesis class of the Scheme 1 booster. By the 


results of the previous section, we have the following: 


[Qual << 2\Qijal + 1)(Hayal + 2) 
|Qijal_ << 2((Qo| + 1)(\Ho| + 2) 


The hypotheses in 7j;4 are majority functions of up to k, hypotheses from Hp). The number 
of unique majority functions of ¢ hypotheses from Ho is given by (I7olti 1) and therefore the 
number of unique majority functions of up to k,; hypotheses from Hp is given by yk (I7eolH 1), 


Using the techniques of the previous section, we can simplify this expression as follows: 


k . 
1 /|Hol +iscl 
pol = (Peet ®") 


i=1 t 


ky . 
= yo (Mer) 1 
t 


i=0 


7 Mol +41) 
= ( k J 7 


([Ho| + 1)" 1 


lA 


Combining these results, we obtain the following: 
| Qin] < 2 (2(|Qo| + L)(|Po| +2) + 1) (Hol + 1) + 1)" 


The complexity of simulating an SQ algorithm depends on log |Qi—|. We have effectively shown 
the following: 
log |Qup| = O(log |Qo| + hike log |Ho]) 
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A.2 Proofs Involving VC-Dimension 


In this section of the appendix, we prove a number of technical lemmas which involve the concept 
of Vapnik-Chervonenkis dimension [36]. We begin by defining VC-dimension and introducing 


a number of preliminary results. 


A.2.1 Preliminaries 


Let G be a set of {0,1}-valued functions defined over a domain X. For any countable set S = 
{@1,.-.-,%m}C X and function g € G, g defines a labelling of S as follows: (g(@1),...,g(@m))- 
5S is said to be shattered by G if S can be labelled in all possible 2” ways by functions in G. 
The VC-dimension of G, VC(G@), is defined to be the cardinality of the largest shattered set. 

VC-dimension is often defined in terms of set-theoretic notation. One can view a function 
g € G as an indicator function for a set X, C X where X, = {a € X : g(a) = 1}. For any set 
SCX, let Ig(S) = {SN X,:9¢ EG}. One can view IIg($) as the set of subsets of S “picked 
out” by functions in G. Note that if IIg(S') = 2°, then $ is shattered by G. For any integer 
m > 1, let Ig(m) = max {|IIg($)|: 5 C X,|S| =m}. One can view Ig(m) as the maximum 
number of subsets of any set of size m “picked out” by functions in G. Note that if Ig(m) = 2”, 
then there exists a set of size m shattered by G. One may define VC-dimension in terms of 
IIg(m) as follows: VC(G) = max {m: II[g(m) = 2™}. 

We next prove a lemma concerning IIg(m) which is used extensively in the sections that 


follow. 
Lemma 6 [f G = G,; UGo, then IIg(m) < Ig,(m) + IIg,(m). 


Proof: For any m, let $,, be a set of size m such that |IIg(5,,)| = II[g(m). Note that such a 
set is guaranteed to exist by the definition of Ig(m). We next note that Ig(S,,) = Hg,(Sin) 
IIg,(Sin), and therefore |IIg(5im)| < [Ilg,C5in)| + |Ile,(5in)|. The proof is completed by noting 
that |I¢,(5m)| < Hg, (mm) and |I¢,(5m)| < He,(m). 


¢ 


The growth of the I[g(m) function plays an important role in proving a number of results 
in PAC learning. Note that for any m < VC(G), Ig(m) = 2”. The following result due to 
Sauer [27] upper bounds the growth of IIg(m) for all m > VC(G). 
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Lemma 7 (Sauer’s Lemma) Let G be a set of {0,1}-valued functions, and let d = VC(G). 
d 

For all integers m > d, IIg(m) < ¥ ("). 
t=0 


Blumer et al. [7] have shown that for all integers m > d > 1, 74, (™) < (em/d)* where e 


is the base of the natural logarithm. We present a new and simpler proof of this result below. 
d 

Lemma 8 For all integers m >d>1, Y (") < (em/d)4 
i=0 


Proof: Since 0 < d/m < 1, we have: 


VEO) «290 


lA 


IA 
Mea 
o— 
J 
— 
|= 


Dividing both sides of this inequality by (d/m)* yields the desired result. 


We may now characterize the growth of the IIg(m) function as follows: IIg(m) grows expo- 
nentially up to m = VC(G@), and II[g(m) grows at most polynomially after that. We may use 
this fact to obtain an upper bound on the VC-dimension of G in the following way. Suppose that 
for some m, we could show that IIg(m) < 2” . Then m must be larger than the VC-dimension 


of G. 


A.2.2 VC-Dimension of Q’=QUQ 


We now prove a result used in Section 3.3 concerning the VC-dimension of the query space used 
by our simulation of an SQ algorithm in the classification noise model. Recall that X is our 
instance space, and Y = X x {0,1} is our labelled example space. For any labelled example 
y = (2,1), we define 7 = (a,1), and for any query y, we define Y(y) = x(¥). Finally, for any set 
of queries Q, we define O = {Y: y € Q}. 
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If Q is the query space of an SQ algorithm, then Q’ = QU Q is the query space of our 


simulation of this SQ algorithm. We may bound the VC-dimension of Q’ as follows. 


Lemma 9 /f QO’ = QUQ, then VC(Q') < «- VC(Q) for a constant c © 4.66438. 


Proof: We first claim that VC(Q) = VC(Q). This fact can be shown as follows. For any 
x € O, we have that y(y) = Y(y). Note that if y € Q, then Y € Q. For any countable set 
T = {y1,---5Ym}, the labelling of T induced by y is identical to the labelling of T induced 
by Y where T = {7,,...,9,,}. Therefore, if there exits a set of size m shattered by Q, then 
there exists a set of size m shattered by Q. This implies that VC(Q) > VC(Q). The fact that 
VC(Q) > VC(Q) is shown similarly, and thus VC(Q) = VC(Q). 

Let d= VC(Q) = VC(Q). For any m > d, we have both Ig(m) < (em/d)* and I5(m) < 
(em/d)*. Thus, for any m > d, we have IIg(m) < Ig(m) + IIg(m) < 2(em/d)*. 


If Igi(m) < 2” for some m, then m > VC(Q'). Thus, any m > d which satisfies 
2(em/d)* < 2” 


is an upper bound on VC(Q’). Setting m = c-d and solving for c, we obtain: 


2(ecd/d)? < 24 
> 2(ec)? < (2°)4 
< (2ec)? < (2°)4 
> 2ee < 2° 
< c > 4.66438 


Thus, VC(Q') < «- VC(Q) for a constant ¢ & 4.66438 


A.2.3. The VC-Dimension of the Query Space of Boosting 


We now prove a result used in Section 3.1 concerning the VC-dimension of the query space of 
boosting. Let Qj and Ho be the query space and hypothesis class used by a weak SQ learning 
algorithm. The queries used by the strong SQ learning algorithm obtained by either Scheme 1 
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or Scheme 2 boosting are of the form x, xj and XAX} where y € Q, and xj is constructed from 
hypotheses in Ho. 

A particular query xj is defined by ¢ hypotheses and an integer 7,0 <j <1. x} (#0) is 1 
if exactly 7 of the 7 hypotheses map « to J, and xi (a, 1) is 0 otherwise. Note that 7 is bounded 
by ky = za lnt in Scheme 1 boosting, and 7 is bounded by ky = saint in Scheme 2 boosting. 
Also note that the hypotheses used to construct a particular xj need not be distinct. 


For fixed 7 and 7, let , ‘ be the set of all xj queries. In addition, we make the following two 


definitions: 


U.; 
j=o 
k . 
i=1 
For any two sets of {0,1}-valued functions .A and B, we define 
ANB= {fa \ fo : fa EA, fs € B}. 
The query space of boosting, Qg, may then be given as follows: 
9p =U," UGA, 


Note that & = k, in the case of Scheme 1 boosting, and & = ky in the case of Scheme 2 boosting. 

We may bound the VC-dimension of Qz, in terms of the VC-dimensions of Q) and Ho in a 
manner similar to that used in the previous section. In particular, we bound IIg,(m), IHpm(m) 
and Ig,,rvui(m). By applying Lemma 6, we obtain a bound on Ilg,(m). From this bound, we 
obtain a bound on the VC-dimension of Qg. We begin by examining , ‘. 


For any hypothesis h : X — {0,1}, we define h: Xx {0,1} = {0,1} as follows 
h(a,1) = (h(x) =1) 


where = is the binary equivalence operator. Thus, h(a, 1) is true if and only if the hypothesis 


h maps x to l. Let Hy = {h :h € Ho}. We may now define a query xj € ,* as follows. Let 


7g 
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hy,...,h; be the 2 hypotheses used to construct xj. 


1 if exactly 7 of hy(a,l),...,hi(a,1) are 1 
xi(e,) = 
0 otherwise 


From a set-theoretic perspective, we can view xj and f as indicator functions for subsets of 


Y = X x {0,1}. We then have the following: 
Y= ty € Y :y is an element of exactly 7 of the sets Y¥;,...,Y;,} 


We next relate I[pi(m) with ILg (m) as follows. 


IL- Lol 
Claim 5 IIpi(m) < ( iig(™) + i ) 
i i 


Proof: Consider a particular xj. We can view xj as either a mapping from Y to {0,1} or as an 
indicator function for a set Yyi CY. In the discussion that follows, it will be more convenient 
to view xj as an indicator function. 

Let T be any subset of Y of size m. Ig (T) is the set of subsets of 7 picked out by 
functions h € Ho, and Tp: (T) is the set of subsets of T picked out by functions xj in, ‘. By the 
definition of Xj note that each unique set in Tp: (7) must correspond to a unique collection of 2 
sets in Ig (T). However, the 7 sets in each unique collection need not be distinct. Therefore, 
the number of unique collections is given by the number of arrangements of 7 indistinguishable 
balls in |ILg (7)| bins. We thus have 

[We (P)|+iel 
MrT] x (REET) 


v 


which implies the desired result. 


By Lemma 6 and the definition of ,*, we now have 


H(m) < (G4 1)- (Taal) + i 1) | 
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Furthermore, by Lemma 6 and the definition of , !*!, we have 


Tpui(m) < < Sit 1): (Talim teeny, 


i=1 
By applying the well known fact that ILa,s(m) < Ia(m)- I[g(m) [4, pg. 104], we now have 
k 
Ils +761 
Ig, arm(m) )< Ilg,(m m) >~( a + 1) ml 7, (m ). 
i=1 


Finally, by Lemma 6 and the definition of Og, we have 


Ilo, (m) S 


Ho,(m) + S7(i +1): (% ,(m mee traslin i+): (OR .(m mare, (A.4) 


i=l t=1 a 
Note that Equation A.4 is of the same form as Equation A.1. We can therefore simplify 


Equation A.4 in a similar manner to obtain: 


IIg,,(m) < 2(Mg,(m) + 1(ILg,(rm) + 2)" (A.5) 


In order to bound the VC-dimension of Og, we must relate the VC-dimension of Ho with 


the VC-dimension of 7. 
Claim 6 VC(Ho) = VC(Ho) 


Proof: We begin by noting that for any instance « € X, h(x) = 1 if and only if h(a, 1lj=1. 
For any countable set S = {21,...,%m} and hypothesis h € Ho, the labelling of $ induced by h 
is identical to the labelling of T induced by h where T = {(v1,1),...,(@m,1)}. Thus, if there 
exists a set of size m shattered by Ho, then there exists a set of size m shattered by Ho. This 
implies that VC(Ho) > VC(Ho). 

We next note that for all functions h € Ho, h(a, 1) = 7h(x,1). Now let 


T = {(xishi),. 0.5 (ms ln) } 


be any countable set shattered by Ho. If (2,1) € T, then («,1) ¢ T since (,1) and («,T) cannot 
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be labelled identically, which is required for shattering. Thus, $ = {a1,..., a} is of size m. 

Now note that h(a) = 6 if and only if h(2,1) = (b= 1). Consider any labelling (6;,...,m) 
of S. This labelling of S would be induced by the hypothesis h € Hy corresponding to the 
function h € Ho which labels T as follows: ((b, = l,),...,(bm =/m)). Since T is shattered by 
Ho, such a function and corresponding hypothesis must exist. Thus, if there exists a set of 
size m shattered by Ho, then there exists a set of size m shattered by Hy. This implies that 
VC(Ho) > VC(Ho). 


We are now in a position to prove the main result of this section. 
Lemma 10 VC(Qz) = O( VC(Qo) + VC(Ho) - k log k) 


Proof: In order to bound the VC-dimension of Qg, we need only find an m which satisfies 
Ilg,,(m) < 2”. We begin by further simplifying the expression for IIg,,(m). 
Assume that Ig,(m) > 1 and Ig (m) > 2. Each of these assumptions is assured when 


m > 1 and the VC-dimensions of Q, and Ho are at least 1. We then have the following: 


Hg,(m) < 2IHg,(m) + 1)(Ig,(m) +2)8 
< _ 22lg,(m))(21g,(m))* 
= 2**? Tlg,(m) (IL g(m))* 


Let go = VC(Qo) and let do = VC(Ho) = VC(Ho). For any m > max {qo, do}, we have both 


IIg,(m) < (em/qo)* and Ig (m) < (em/do)*. We now have: 
IIg,,(m) < 2°**(em/qo)”(em/do)* 


To bound the VC-dimension of Qg, we need only find an m which guarantees that the right- 


hand side of the above inequality is at most 2™. 


2**?(em/qo)"(em/do)** 


lA 


om 


=  (k+2) + qola(em/qo) + dok lg(em/do) 


IA 
3 


(A.6) 


For fixed do, qo and k, the above inequality has the form m > gi(m) + g2(m) + g3(m) where 
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each function g;(m) “grows” more slowly than m. In particular, each function g; satisfies the 
following property (recall that we are restricted to values m > max {qo,do}): If mz; > gi(3m;), 
then m > g;(3m) for all m > m;. Our strategy is as follows. Find appropriate values of m; 
which satisfy m; > g;(3m;), and let m = 3max{m,,m2,m3}. Then m must satisfy m > 
gi(m) + go(m) + gs(m). The reasoning is as follows. Suppose, without loss of generality, that 
m, = max{m,,Mm2,m3}. We then have m = 3m,. Furthermore, m, > gi(3m,), and since 
m, > mz and m, > ms, we also have m, > g2o(3m,) and m, > gs(3m,). Combining these 
inequalities, we have 3m, > gi(3m1) + go(3m1) + g3(3m,1) which implies the desired result. 
For gi(m) = k + 2, we may simply choose m, = k + 2. For go(m) = qolg(em/qo), we chose 


M2 = 6qo, which is verified as follows: 


IV 


qo lg(e(3 + 6qo)/qo) 


Ig(18e) © 5.613 


640 


V 


a 6 


For g3(m) = dok lg(em/do), we choose m3 = 9dok lg k, which is verified as follows: 


Qdoklgk > doklg(e(3-9dok lg k)/do) 
cs 9lgk > Ig(27eklgk) 
> ko > 27eklgk 
< k’ > 2e 


This final inequality is true for any & > 2. We have effectively shown the following, which 


completes the proof: m = 3 max {k + 2, 64q0, 9dok lg k} = O( qo + dok log k). 


A.2.4 The VC-Dimension of the Query Space of Hybrid Boosting 


We now prove a result used in Section 3.1 concerning the VC-dimension of the query space of 
hybrid boosting. As in Section A.1.2, let On, be the query space of the hybrid booster, and let 


Qi/4 and 71/4 be the query space and hypothesis class of the Scheme 1 booster. Furthermore, 
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let ky = ky(y,1/4) and ky = ko(1/4,€). We then have the following analogs of Equation A.5: 


Hg,,(m) < 2(IHo,,,(m) + L)(Ig, ,(m) + 2)" 


Ig,,,(m) < 2(Ig,(m) + 1), (mm) +2)" 


Now, 74/4 is the set of hypotheses which are majority functions of up to k, hypotheses from 
Ho, and Hiya = {h :h€ His}. However, one can also define each function he Hiya as 
follows. Given a function h € Hiya, h corresponds to some hypothesis h € Hy,/4 which in turn 
corresponds to some set of hypotheses {h,,...,h;} from Hp where j < k,. By definition, we 


have 


h(a,l) = (maj {hi(a),...,hj(@)} = 0). 


However, it is also the case that 


(maj {h,(«),...,h;(2)} =1) = maj {hy(2,1),...,h;(#,D}. 


Thus, we can think of Hiya as the set of majority functions of up to &, functions from Ho. 
Now, let He 4 be the set of majority functions of 7 functions from Ho, and let T be any 

subset of Y of size m. Ig (T) is the set of subsets of 7 picked out by functions he Ho, and 

IL- 


gi (T) is the set of subsets of T picked out by functions he Fy. Note that each unique 
1/4 

set in Wye (7) must correspond to a unique collection of j sets in lg (T'). Since the 7 sets 

in each unique collection need not be distinct, the number of such unique collections is given 

by the number of arrangements of j indistinguishable balls in [Ig (T)| bins. Thus, we have 


[ai (T)| < (Mi. IH) which implies that 
1/4 j 


~ ki oA. 
Since Hija = U Hi), by Lemma 6, we have: 
j=l 


k . 

> He (m)+ jel 
Ty, 7) < ( * } ) 

j=l 


Jj 
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j=o j 
- (Tae) t *) 21 
ky 
< (Ig(m)+ 1)" 1 


Combining the above results, we have the following: 
k k Ke 
Honn(™m) < 2 (2(Mg.(m) + 1(Mg,(m) +2)" +1) (Ig, (m) + 1)" +1) 


Now, assume that IHg,(m) > 1 and I[g (m) > 2. Each of these assumptions is assured when 


m > 1 and the VC-dimensions of Q, and Ho are at least 1. We then have the following: 


A 


& 41) ((21Lg,(m))# + 1)” 
*) (2!"ULg, (mye) 


= HDS TT 9, (an) (Lg, (sn) OED 


A 


2 (2(2Ig,(m))(2Ig (m)) 
(m)) 


2 (2* 911g, (m)(II 


Let go = VC(Qo) and let do = VC(Ho) = VC(Ho). For any m > max {qo, do}, we have both 


Ig,(m) < (em/qo)” and Ig (m) < (em/do)®. We now have: 
Tapa (m) < 2 tVEtU+3 (em / qa)! (em/dy Ort) 


To bound the VC-dimension of Qyp, we need only find an m which guarantees that the right- 


hand side of the above inequality is at most 2”. 


2HADEAD+3 (em / gy)! (em [dy oe h2t) gm 


lA 


= ((Ay + 1)(Ko + 1) + 3) + golg(em/qo) + doki(ko + L)lg(em/do) < m (A.7) 


Inequality A.7 has the same form as Inequality A.6. By appropriate substitution, we find that 
m3 max{(ky + 1)(ke + 1) + 3, 6q0, 9doki (ke + 1) lg(ki(ke + 1))} = O((qo + doky ke log(k,k2)) is 


sufficient to satisfy the above inequality. We have effectively proven the following: 
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Lemma 11 VC(Qye) = O( VC(Q9) + VC(Ho) - kiks log(kyk2)) 


A.3 A Lower Bound for Probabilistic SQ Algorithms 


Throughout this thesis, we have assumed that SQ algorithms are deterministic and output an 
accurate hypothesis with certainty. In this section of the appendix, we relax this condition by 
allowing probabilistic SQ algorithms which output accurate hypotheses with high probability. 
In particular, we show a lower bound on the query and tolerance complexities of such SQ 
algorithms which is analogous to the result obtained in Section 3.2.2. 

Consider the two player game described in Section 3.2.2. We modify this game by allowing 
the player to be probabilistic, and we only require that the player output an acceptable set 


with probability at least 1 <6, for some some 6 > 0. We may now show the following. 


Lemma 12 For anyd>4,t<d/4,6< 1/8 and N = Q(d'**) for some a > 0, the probabilistic 


player requires Usrtew) queries of the oracle. 


Proof: By incorporating techniques from Kearns’ lower bound proof [19], we can modify 
the original proof of Lemma 1 as follows. The adversary chooses the target set S randomly 
and uniformly from the set So of all (*) subsets of [.V] of size d. Consider the first query, Q:, 
submitted by the player. Q, partitions Sp into d+ 1 sets S$, S},...,S4 where each subset $ € S$} 
has |S 1 Q,| = 7. Since the choice of the target set was random, uniform and independent of Q,, 
the probability that the target set is an element of S} is proportional to |S}|. Note that S,, by 
definition, is the set Si of which the target is a member. 

For any k > 2, consider all Si for which |Si] < |So|/(k-(d+1)). Since there are only d+1 


sets, the total cardinality of such “small” sets is less than |So|/k/. Thus, we have the following: 


Pr|isil> z az | Bol lh = 1 e/h 


-(d+1) |So| 


By successively applying this result through & queries, we obtain the following: 


|So| k 
Pr Is > acesna > (11/k) 
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Note that for any k > 2, (1 41/k)* € [1/4,1/e). Thus, with probability at least 1/4, we have 
a lower bound on the size of |S;,|. We next show that if |S;| is sufficiently large, then there is a 
significant probability that the player will fail if it halts and outputs a set at this point. 

Let T be any set output by the player at the end of the game. For any 7,0 <i < N, note 
that there are exactly (‘Y) sets S$ € 2!) such that |S A T| = i. Thus, there are exactly (*)) sets 
5 € 20] such that |S A T| < t. Now suppose that |S,| > 2(%)). Since the target set is equally 
likely to be any element of S;, the probability that T is an acceptable set is at most 1/2. Thus, 
the player will fail with probability at least 1/8 if it halts after & questions for any k which 


satisfies the following inequality: 


(Qed) <athg mess 


Solving the third inequality for (k-(d+1))* and noting that d > 4, we have the following: 


wus = HG AY 


N 3d/4 
< k-(d+1))*¥ < {— 
(k-(d+1))¥ < (=) 
3d N 
<< — 
cs klogk+klog(d+1) < ~* ios (<5) 


The latter inequality is implied by the following two inequalities: 


3d N 


3d N 
8 log ded 


log( log 5) 


A 


3d N 
< a 
klog(d+1) < 5 log (=) 
e p< 28a 
log(d + 1) 
Each of these inequalities is implied by 
_ Slog & _ ( dlog N ) 
log((d + 1) log 45) log(dlog NV) 
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for N = Q(d'**). 


Combining this result with the proof of Theorem 5, we immediately obtain the following: 


Theorem 21 There exists a parameterized family of function classes which require OE) 


queries with additive error O(€) to learn in the SQ model by a probabilistic algorithm. 


Part II 


Searching in the Presence of Errors 
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CHAPTER 7 


Introduction 


Coping with errors during computation has been a subject of long-standing interest. It has 
motivated research in such areas as error-correcting codes, fault-tolerant networks, boolean 
circuit evaluation with faulty gates, and learning in the presence of errors. In the following 


chapters, we focus on the problem of searching in the presence of errors. 


7.1 Introduction 


Our goal is to find an unknown quantity x in a previously specified, discrete, but not necessarily 
finite, domain by asking “yes-no” questions, when some questions are answered incorrectly. 
We show that it is possible to cope with errors whose number may grow linearly with the 
number of questions asked, and, depending on the class of questions allowed, to do so with an 
asymptotically optimal number of questions. Examining both adversarial and random errors, 
we find that even in a fairly restricted adversarial error model, searching is at least as difficult 
as in the random error model. 


The problem can be further qualified by: 
e Kinds of questions that may be asked. 


— Comparison questions: “Is x less than y?” 
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— Membership questions: “Is z in the set $?”, where S is some subset of the 


domain. 
e Kinds of errors possible. 


— Constant number: It is known a priori that there will be at most k errors, where 


k is some fixed constant. 


— Probabilistic: The answer to each question is erroneous independently with some 
probability p,O<p< S. 

— Linearly Bounded: For some constant r, 0 < r < :; any initial sequence of 2 
answers has at most rz errors. This model allows the answers to be erroneous in a 
malicious way. Unlikely scenarios in the probabilistic model (such as a long sequence 


of correct answers followed by a short sequence of false ones) must be dealt with here. 
e Domain of the quantity being sought. 


— Bounded: z¢ € {1,...,n}, for some known n. 


— Unbounded: x may be any positive integer. 


Much research has been devoted to the subject of searching in the presence of errors. 
Rivest et al. [25] have shown that in the bounded domain with at most k errors, « can be 
determined exactly with lgn + klglgn + O(klgk) comparison questions.1 Here k can be a 
function of n, but not of the number of questions asked. When & is a constant, this is an 
asymptotically optimal bound since [lg n] is a lower bound on the number of questions needed 
to search even without errors. Naturally, this bound also applies to searching with membership 
questions, since comparison questions are a restricted version of membership questions. 

In the probabilistic error model, where errors occur randomly and independently with prob- 
ability p, we must find the correct x with probability of failure at most 6. Since 6 is previously 
known and fixed, we consider 6 a constant for the purpose of measuring the complexity of the 
searching algorithm.” Pelc [24] showed that in the probabilistic error model, with error proba- 
bility p < 1/2, O(lg* n) questions are sufficient to search in the bounded domain. Frazier [13] 


'The term lg n denotes log, x throughout this thesis. 
? Typically, the complexity of such algorithms depends on log(1/5), as does the complexity of our algorithm. 
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Membership Questions Comparison Questions 
o<r<t | E<r<dt floxr<t | tx<r<t 


Pele | odgn) | ow'®=™) | ofgn) | o's E77) 


Thesis O(lgn) O(n'8 TF) 


Figure 7.1: Bounds for searching in the bounded domain with linearly bounded errors. Here 
nis a bound on the number being sought. 


improved the bound to O(lg n lglg n) questions using a somewhat complicated analysis. Finally, 
using standard Chernoff bound techniques, Feige et al. [12] showed that O(lgn) questions are 
sufficient for any p < 1/2. Our contribution here is a formal reduction from the problem of 
searching in the probabilistic error model to that of searching in the linearly bounded error 
model. To state this result informally, we show that an algorithm for searching in the presence 
of linearly bounded errors can be transformed into an algorithm for searching in the presence 
of random errors. In this sense, searching with linearly bounded errors is at least as difficult as 
searching with random errors. When we are allowed to ask membership questions, this reduc- 
tion together with the results from the linearly bounded error model mentioned below matches 
the Feige et al. O(lgn) bound in the bounded domain. We also generalize this bound to the 
unbounded domain.® 

In the linearly bounded error model, Pelc [24] showed that x can be determined exactly in 
O(lgn) questions in both the bounded and unbounded domains. However, these bounds only 
hold for r < 1/3. The best known bound using comparison or membership questions in the 


1 


bounded domain for 1/3 < r < 1/2 was O(n'§™=*). Note that the degree of the polynomial 


in this bound is unbounded as r approaches 1/2. This bound comes from an analysis of a 
“brute-force” binary search, where each question of the search is asked enough times so that 
the correct answer can be determined by majority. A simple argument [13, 32] shows that the 
search problem cannot be solved (with either membership or comparison questions) if r > 1/2. 

We show significantly improved bounds in the linearly bounded error model which hold for 
the entire range 0 < r < 1/2. With memberships questions, we show that x can be determined 


exactly in O(lg) questions in both the bounded and unbounded domains. These bounds are 


°In the unbounded domain, n now refers to the unknown number. 
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Membership Questions Comparison Questions 
o<r<t | E<r<dt floxr<t | tx<r<t 
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Figure 7.2: Bounds for searching in the unbounded domain with linearly bounded errors. Here 
n is the unknown number. 


tight since searching has a trivial Q(lgn) lower bound. With comparison questions, we improve 


1 


the bounds to O(n'8 =) = o(n) questions for the bounded domain and O([nlg? n]8 =) = o(n) 


in the unbounded domain. A comparison of this work with the best known previous results 
can be found in Figures 7.1 and 7.2. Our results are obtained by looking at the search problem 
in the framework of chip games. These chip games have also proved useful in modeling a 
hypergraph 2-coloring problem [5]. In general, chip games model computational problems in 
such a way that winning strategies for the players translate into bounds on the critical resource. 
This critical resource is represented by some aspect of the chip game, such as number of chips 
used or number of moves in the game. 

Spencer and Winkler [32] have also examined this problem. They have arrived independently 
at one of the theorems in this paper using different proof techniques. Their paper as well as 
one by Dhagat, Gdcs, and Winkler [10] considers another linearly bounded model of errors. 

We begin in Section 7.2 by developing the framework of chip games within which we solve the 
search problem. Chapter 8 begins with a simple strategy for solving our problem in the linearly 
bounded model in the bounded domain which works with either comparison or membership 
questions, but whose obvious analysis gives an inefficient bound on the number of questions. 
We then improve this bound by analyzing this strategy using chip games. Chapter 8 continues 
by focusing on membership questions and proving an O(lg 7) question bound for this class. The 
chapter ends with a generalization of the above bounds for the unbounded domain. Chapter 9 
contains the aforementioned reduction between the probabilistic and linearly bounded error 
models in the bounded domain, and the O(lgn) question bound for the probabilistic error 
model which follows from it. These results are also generalized to the unbounded domain. 


Chapter 10 concludes the paper with a summary of the results and mention of some open 
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=<—— boundary line 


Figure 7.3: Chip Game 


problems. 


7.2 Searching and Games 


Searching for an unknown number z in {1,...,n} by asking “yes-no” questions can be restated 
in terms of the game of “Twenty Questions”. In this game between two players, whom we 
denote Paul and Carole,* Carole thinks of a number between 1 and n. Paul must guess this 
number after asking some number of “yes-no” questions which is previously fixed. Our goal 
in this game is to determine how many questions Paul must be allowed in order for him to 
have a winning strategy in the game. Clearly, [lgn] questions are sufficient if Carole always 
answers truthfully. The problem of searching with errors thus translates into playing “Twenty 
Questions” with a liar [33]. Corresponding to the aforementioned error models, we consider 
both a probabilistic and an adversarial linearly bounded liar. 

The game against a linearly bounded liar can now be further reformulated as a Chip Game 
between two players: the Pusher and the Chooser. Pusher-Chooser games were first used by 
Spencer [31] to solve a different problem in his notes on the probabilistic method. The Chip 
Game starts with a unidimensional board marked in levels from 0 on upwards (see Figure 7.3). 
We start with n chips on level 0, each chip representing one number in {1,...,n}. At each 
step, the Pusher selects some chips from the board. These chips correspond to the subset S of 
{1,...,n} that Paul wants to ask about. In other words, selecting $ is tantamount to asking 
“Is a € $?”. The Chooser then either moves the set of chips picked by the Pusher to the next 


level, indicating a “no” answer from Carole (a is not in $), or it moves the set of chips not 


“An anagram for the word “oracle,” as this is her role in the game. 
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picked by the Pusher, indicating a “yes” answer from Carole. Therefore, a chip representing 
the number y is moved to the right if and only if Carole says that y is not the answer. The 
presence of a chip representing the number y at level ¢ says that if y is the unknown number 
x, then there have been 2 lies in the game. After some & steps, if a chip is at any level greater 
than |rk|, then it may be thrown away since the corresponding number cannot possibly be the 
answer (too many lies will have occurred). To win, the Pusher must eliminate all but one chip 
from the board. 

To clarify which chips may be thrown away, we maintain a boundary line on the board. 
After & steps, the boundary line will be at level |rké|. Thus the Pusher may dispose of the chips 
at levels to the right of the boundary line at any time. Note that the boundary line moves one 
level to the right after approximately 1/r steps. The number of questions that we need to ask 
to determine x exactly is the same as the number of steps needed for the Pusher to win the 


above Chip Game. 


CHAPTER 8 


The Linearly Bounded Error Model 


In this chapter, we show an O(n'8 =) question bound for searching with comparison questions 
and an O(lgn) question bound for searching with membership questions. We first show an 
Q(n'8 ==) lower bound for a “brute-force” strategy. Strategies similar to this “brute-force” 
method are given by Pelc [24] and Frazier [13], and these were the best known results for 


1/3 <r < 1/2 prior to this work. 


8.1 A Brute-Force Strategy 


To determine an unknown number z € {1,...,n}, a “brute-force” strategy simply performs a 
binary search, repeating each question enough times so that majority gives the correct answer. 
Let q(2) be the number of times question 7 is repeated, and let Q(z) be the total number of 
queries through question i (Q(2) = oat q(j)). To guarantee that majority gives the correct 
answer, we insure that the number of lies the malicious oracle can tell is less than half the 


number of times question & is repeated. We thus obtain the following: 


rQ(h 1) + q(k)) <  q(hk)/2 
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We now use the fact that Q(k) = Q(k $1) + q(k): 


Q(k) = Q(ke1)+q(k) 
2r 


> Q(k 1) + Tay, Ok @1) 
1 
= Ta Gk ed 


Thus, Q(k) = Q((—)*). Since the correct answers to {lg n] binary search questions must be 


obtained, we obtain the following lower bound for the “brute-force” strategy: 


Q(T) = QUE") 


1-2r 1-2r 


A similar upper bound can also be shown for this strategy. While this strategy is sound, its 
naive analysis yields an unsatisfactory bound. We can improve on this significantly through 


the use of chip games. 


8.2 Searching with Comparison Questions 


We now consider an essentially identical strategy in the chip game. The Pusher plays this game 
in phases. Each phase corresponds to getting the correct answer to a single question. At the 
beginning of each phase there is a single stack of chips somewhere on the board. The Pusher 
picks a subset of these chips, generally some half of them which corresponds to a comparison 
question whose correct answer he wishes to determine. He continues picking the same half-stack 
throughout this phase until either it or the other half-stack moves beyond the boundary line. 
Then he begins the next phase with the remaining half-stack. This continues until there is only 
one chip remaining on the board to the left of the boundary line. Note that if there are m chips 
on the board initially, [lg m] questions need to be answered correctly. 

Now consider the board before and after some phase 7. At the beginning of phase 7, there 
is a stack of chips at a level some distance /; away from the boundary line (see Figure 8.1). At 
the end of phase 7, one half-stack has moved some distance 2 from its original position and the 


other half-stack has moved one position past the boundary line. The boundary line is now at 
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Before phase j After phase j 
lj J 
jt+l 


Figure 8.1: Chips before and after phase 7. 


some distance /;,, from the first half-stack. 
Let T(d,1) be the number of steps the Pusher takes to have d questions answered correctly 
when a single stack of chips on the board is a distance / away from the boundary line. We then 
have 
T(d,l;) = T(d $1, 1,41) + (steps during phase 7). 
We next bound the number of steps in phase 7 as a function of r, J; and 1,41. 
Lemma 13 The total number of steps during phase j is less than 


Botta +38 
1-2r : 


Proof: The total number of steps during phase 7 is equal to the total number of levels the two 
half-stacks move. One half-stack moves 7 levels and the other moves 7+/;4; +1 levels, and thus 
the total number of steps in phase 7 is 22 +/,4, +1. 

Let s; be the total number of steps prior to phase 7, and let p,; be the position of the 
boundary line prior to phase 7. Note that p; = [rs;|. Using the fact that «1 < |z| < x, we 


have the following: 


Pyar SS 18544 


pi > 7s; Sl 
Subtracting these two inequalities, we obtain: 
Djs Di < 1 Sj41 8j) +1 


Now, s;41 4s; is the total number of steps during phase 7, and p;4; =p; is the number of levels 
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the boundary line moves. We therefore have the following inequality 
t+ hye) < rQi¢4a+¢H4+1 (8.1) 


which implies that 
lL; “(1 Sryljqi + 1 + r 


L< 
‘ 1 S2r 


Substituting this bound on 2 into the expression 27 + 1;,; + 1, we obtain the desired result. 


Now we are ready to show that: 


O(lo()?) ify > 0 


OU) iflo = 0 


Theorem 22 7T(d,l5) = 


Proof: We first show that Vj, lj41 < ott Solving Inequality 8.1 for J;4,, we obtain: 


i;+14r4 i(2r 1) 
1<sr 
ltr 


lor 


This last inequality holds due to the fact that 2r 61 < 0 andi > 0. Successively applying this 


inequality, we obtain the following: 


_ lp itr ( at 
~  (Ler)i r ler 
1 1 \? 1 
(6+) (ae) oS 
r 1<sr r 


We can now use this fact to obtain a bound on T(d, Ip): 


2ip 34,43 
T(d,l Tideljy-oeee 
(dbo) < Tdelh) + 1S2r 


2g 84,43 2461443 _ 21g, Ola +3 
1<2r 1<2r 1<2r 
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f—) 
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a 
bo 


Toor lai + 3a] 


1 = +P 
S 1<2r [2 3d ¥ | (0+ r \Gye aul 


1 1 ltr 1 \%} lt+r 
= | 3d ~ (104 ; )(() -) “(d1) i 


For any constant r, this expression is O([p(+.)“) if lp > 0 or O((+)") if Ip = 0. 


This result will be used throughout this thesis. In particular, consider the problem of 
searching in the bounded domain with comparison questions. The corresponding chip game 
begins with n chips and the boundary line at level 0. Using binary search, we require the 


correct answers to [lgn] questions. Employing Theorem 22, we immediately obtain: 


Theorem 23 The problem of searching in the linearly bounded error model in the bounded 
domain {1,...,n} with comparison questions and error constant r, 0 <r < 1/2, can be solved 


with O(n'& ==) questions. 


8.3. Searching with Membership Questions 


We show a winning strategy for the Pusher which requires O(lgn) steps. The strategy works in 
three stages. In Stage 1, the Pusher eliminates all but O(lg) chips from the board in O(lg 7) 
steps. In Stage 2, the Pusher eliminates all but O(1) chips from the board in an additional 
O(lgn) steps. In Stage 3, the Pusher removes all but one chip from the board in the final 
O(lgn) steps. 


8.3.1 Stage 1 


The strategy employed during Stage | is simple. We describe it inductively on the number of 
steps as follows. Let h,,(7) be the height of the stack of chips at level i after m steps. In the 
(m+ 1)-st step, the Pusher picks S| from each stack of chips at all levels 7. He continues 
this way for c; lg n steps (where c; is a constant that will be determined in the analysis). 
Before we can analyze this strategy, we will need a few definitions. Define normalized 


binomial coefficients b,,(2) as 
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and let 
Amt) = Am(t) Sbm(2). 


The normalized binomial coefficient 6,,(%) will approximate h,,(7), the height of the stack at 
level ¢ after m steps, while A,,(7) will account for any discrepancy. 

In order to analyze the given strategy, we need to be able to determine the number of chips 
which are to left of the boundary line after some number of steps in our strategy. After m steps, 
this is equivalent to )7;<j4m) Am(%) (since r is the rate at which the boundary line moves). This 
sum is difficult to determine exactly. Instead, we will derive an upper bound for it by using the 
fact that i<jmm) Mm(2) = Vie [remy Om(2) + Vie trmy Am(%). In particular, we will show an upper 
bound for 37 ;< {my Am(t)- 

For the strategy given above, we now bound the discrepancy between the actual number of 
chips in any initial set of 7 stacks and the number of chips predicted by the normalized binomial 
coefficients. We will need three lemmas. The first two lemmas handle boundary conditions, 


while the third is required in the proof of the main theorem. 


Lemma 14 (Vm > 0), A,,(0) < 1. 


Proof: The proof is by induction on m. 
e base case: For m = 0, ho(0) = n = b9(0) = AQ (0) = 0. 


e inductive step: Assume A,,_:(0) < 1. We now have the following: 


Rm(O) < [as 


2 
Am 1 
_ m— 1 
_ 7+ 0), f 
= 5g(0)4 Sel) 4! 
< by(0) +1 


Thus, A,,(0) = hm(0) 8n(0) < 1. 
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Proof: 
So him(i) =n = So dn(i) => SOA, (i) = 0 
2=0 2=0 2=0 
j-l . ; 
bn 
Lemma 16 50 by_i(i) + — 1) = Sdn (i). 
2=0 2=0 


Proof: We first note the fact that (,°,) +(;) =(“f"). The proof proceeds as follows: 


: ("= 
£2) 


joi ; j-1 

; bn n mol n (mol 
bin a(t) + 1) a ( )+ ( ) 
: é 


t= 


Theorem 24 (Vm > 0) (Vj < m), So An(2) <j. 


i=0 


Proof: The proof of the theorem is by induction on m. The base case of m = 0 is trivial. In 


the inductive step, we show that if the theorem holds for m <1, then the theorem holds for m. 


The boundary conditions 7 = 0 and 7 = m are handled by Lemmas 14 and 15. We concentrate 


on the case 0 < j < m below. Consider the following (see Figure 8.2): 


M 
a 
a 
IA 


j-l 
: hAm—1(J) 1 
hm—1(2) T + a 


lA 
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After step m-1 After step m 


0 1 2 3 ... j-l j wee O 1 2 3... j-l 
CT - chips that move 


He. chips that do not move Note: shaded chips are the same 


Figure 8.2: Chips before and after step m 


ait 1 
+E dnl (2a. 


We now bound the quantity ie Am-1(4) + Smaitl) + :. There are two cases, depending upon 


whether A,,-1(7) < Ll or An_i(y) > 1. If Am_i(7) < 1, we have the following: 


gol . pod 
4 Am- 1 ‘ 
Anal i( Am=1(9) + i < $2 Am-1(i) + 1 
2 2 7=0 
< j+1 


If A,,-1(j) > 1, then Smaild) +4 <A,,-1(j). We thus obtain the following: 


jr-l . j 
4 Am- 1 : 
2 Am-ald) Seat Sanat) 


We therefore have 
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< 


bm(t) +9 +1 


0 


j 
— 


j 

which implies that So An(2) <j. 
i=0 

Now we will bound S);<jrmy &m(%). We will find a constant ¢, such that for m = ¢ Ign, 


soln b(t) is a constant. If we can do this, then it follows from the theorem above that 
sem! h(i), the number of chips remaining to the left of the boundary line, is O(lgn). The 


reasoning is as follows: 


[rm] [rm] [rm] 
» Am(t) = » b(t) + » Am(%) 
[rm] 
< » b(t) + [rm] +1 
= 4+ (|r-algn| 


for appropriate constants cz and ¢3. 


In order to determine c,;, we make use of the following bound [22]: 


k 
3 (") < 2hH(n) 


i=0 Q 


where 0 <  < 1/2 and H(r) is the binary entropy function.’ We now have: 


[rm| [rm] 
n ™m 
Sal) = a (7) 
2=0 2 2=0 t 
< LomH(r) 
> 5m 
—  pgm(H(r)=1) 


lgn 


This last quantity is O(1) when m = = Toe Thus if we pick c, = then after 


1 


m = c,lgn steps, there will be at most c3lgn chips remaining on the board to the left of the 


'H(r) =—-rlgr —(1—r)lg(1—r) 
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boundary line. The strategy in this stage can also be applied to the game where the boundary 
line starts out at level / = O(lgn) instead of at 1 = 0. One can show directly or through the 
use of the techniques given in Sections 9.1.1 and 9.1.2 that Stage 1 still ends in O(lgn) steps 
with at most O(lgn) chips to the left of the boundary line. This fact will be useful when we 


examine the unbounded domain. 


8.3.2 Stage 2 


At the end of Stage 1 we are left with some cz lg chips on the board with the boundary line 
at level c, lg n (for appropriate constants c, and c.). After Stage 2, there are O(1) chips on the 
board to the left of the boundary line after O(lg nm) additional steps. 

Before starting Stage 2, we alter the board by moving everything on the board (chips and 
boundary line) to the right by c.lgn, so that the boundary line is now at level (c; + cz) lgn = 
clgn. While this new board corresponds to a different game than the one we have played until 
now (it corresponds to a game in which many more questions and lies have occurred), these 
two boards are equivalent in the sense that the Pusher can win from the first board within k 
extra moves if and only if he can win from the second board within & extra moves. 

Now move the chips to the left in such a way that there is exactly one chip on each of the 
first cy lg n levels. Note that the Pusher does not help himself by doing this, since moving chips 
to the left is in effect ignoring potential lies which he has discovered. 

At each step in this stage, the Pusher first orders the chips from left to right, ordering chips 
on the same level arbitrarily. He then picks every other chip according to this order; that is, he 
picks the lst, 3rd, 5th, ... chips. We say that the board is in a nice state if no level has more 


than two chips. 
Lemma 17 Throughout Stage 2, the board is in a nice state. 


Proof: We show this by induction on the number of steps in Stage 2. Certainly at the beginning 
of Stage 2, the board is in a nice state since each level is occupied by at most one chip. Now 
suppose that the board is in a nice state after 7 steps. Consider any level 7 after the (7+ 1)-st 
step. Since both levels 7 1 and j had at most two chips before the (¢ + 1)-st step, after this 


step level 7 retains at most one chip and gains at most one chip, thus ending with at most two 


chips. 
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We now show that after O(lgn) steps, there are at most 2k chips remaining to the left of 
the boundary line. Here & is a constant (depending only on r) which will be determined later. 
If there are fewer than 2 chips to the left of boundary line, Stage 2 terminates. Let the weight 
of a chip be the level it is on, and let the weight of the board be the weight of its 24 leftmost 


chips. 
Lemma 18 After each step in Stage 2, the weight of the board increases by at least k 1. 


Proof: Of the 2k leftmost chips after step 7, at least (2k <1) chips remain in the set of leftmost 
2k chips after step 7+ 1. (The 2k-th chip may be on the same level as the (2k + 1)-st chip. In 
this case, if the 2k-th chip moves in step 7+ 1, then the (2k+ 1)-st chip becomes the new 2k-th 


chip in the revised ordering.) At least || = k<1 of these chips move to the right one level 


during step «+ 1, thus increasing the weight of the board by at least k 1. 


Let S be the number of steps taken during this stage and let W be the weight of the board 
at the end of these $ steps. Since the weight of the board goes up by at least k<1 in each step, 
and since the initial weight of the board was non-negative, W > (k<1)$. At the end of the S$ 
steps, the boundary line is at clgn + |[r5'|. Since this stage ends when there are fewer than 2k 
chips to the left of the boundary line, we certainly have W < 2k(clgn+7r5). Combining these 


two inequalities, we obtain: 


2k(clgn+rS) > S(k1) 
2ke 1 
k 1 2kr 6 


lA 


If we let k = + -, then S < S-lgn = O(lgn). Thus after O(lgn) steps, Stage 2 ends 


1-2r 


leaving at most 2k chips to the left of the boundary line. 


8.3.3 Stage 3 


At the beginning of Stage 3, the Pusher moves all of the remaining chips to level 0. Again this 
is legal, since he is essentially choosing to ignore some information he has gathered. We now 
have some 2k chips a distance clgn away from the boundary line (for appropriate constants c 


and k). By applying Theorem 22 from Section 8.2, the Pusher can win this game in O[(clgn)- 
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(+)l6?'l] = O(lgn) steps. Since each of the three stages takes O(lg n) steps, we now have the 


l-r 


following: 


Theorem 25 The problem of searching in the linearly bounded error model in the bounded 
domain {1,...,n} with membership questions and error constant r, 0<r < 7 can be solved 


with O(lgn) questions. 


8.4 Unbounded Search 


Now consider the problem of searching for a positive integer in the presence of errors as before, 
but where no upper bound on its size is known. Let this unknown integer be n. Using strategies 
developed in this paper already, we show that n can be found with O(lg n) membership questions 


and O([nlg* n]'=*) = o(n) comparison questions. 


The search occurs in two stages. First, we determine a bound for the unknown number n. 


Second, given a bound on n, we employ the techniques for bounded searching given above. 


8.4.1 Unbounded Search with Membership Questions 


Consider the problem of bounding the unknown number n if all of the answers we receive are 
known to be correct. We could ask questions of the form “Is « < 2?'?”. We would begin by 
asking “Is « < 22°?” Tf the answer were “no”, we would follow with “Is x < 2?'?”, and so on. 


Mele x] 


Since n <2 , we will obtain our first “yes” answer (and thus have a bound on n) after at 


most [lglg n] questions. We further note that our bound is not too large: 


o2heienl < g2isients _ o2lgn _ n2 


Employing the techniques and results of Section 8.2, we can use the above strategy in the 
presence of errors. We need the correct answers to [Iglgn] questions. By Theorem 22, we can 


obtain these answers in 
O( A )Ms'8"1) = O((Ig n)8=*) = o(lg n) 


questions. 
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Having found a bound for n, we have reduced our unbounded search problem to a bounded 
search problem. We can now apply our bounded search strategy of Section 8.3. It is important 
to note that since we have already asked o(lgn) questions, the boundary line will have moved 
to o(lgn). But recall that Stage 1 of our bounded search algorithm can tolerate the boundary 
line starting at O(lgn). Thus the Pusher can now start with all relevant chips at level 0 and 
boundary line at level o(lgn) and apply the bounded search strategy of Section 8.3. Since our 
bound on the unknown number n is at most n”, we will finish this stage after O(Ig(n”)) = O(lgn) 


questions. We can now claim the following: 


Theorem 26 The problem of searching in the linearly bounded error model in the unbounded 
domain with membership questions and error constant r, 0 <r < 5, can be solved with O(|gn) 


questions, where the number being sought is n. 


8.4.2. Unbounded Searching with Comparison Questions 


We can employ techniques similar to those used above to solve the unbounded search problem 
using comparison questions. We first determine a bound on the unknown integer n using 
the strategy developed above. We thus bound the unknown number n by at most n? using 


1 


O((lg x) =F) questions. Note that the boundary line will now be at O((lgn)'!@™). 


Having bounded the unknown number n by at most n*, we could simply use Theorem 22 
directly. By performing a simple binary search, we will need correct answers to at most [lg(n*)] 


questions. Using Theorem 22, we obtain an overall question bound of 


This can be improved, however, by adding an extra stage. After bounding the unknown 
number n by at most n”, partition this bounded interval into exponentially growing subintervals 
I; = [2',2/*' 1] Vj > 0. Note that there will be at most [lg(n*)] such subintervals. To 
determine the correct subinterval, we perform a simple binary search on these subintervals 


requiring correct answers to [lg [lg(m”)]] questions. By Theorem 22, we will need 


Of(igmy's mr -(Lylis ls O11) = O({lg? n]'8 =>) 
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additional questions. Since our subintervals grew exponentially, the subinterval containing the 
unknown number n will be of size at most n. We can thus perform a final binary search on this 


subinterval and employ Theorem 22 to obtain an overall question bound of 


Theorem 27 The problem of searching in the linearly bounded error model in the unbounded 
domain with comparison questions and error constant r, 0 <r < 7 can be solved with 


O({n lg? nJ8=*) = o(n) questions, where the number being sought is n. 


CHAPTER Q 


The Probabilistic Error Model 


Recall that in the probabilistic error model, Carole lies randomly and independently with 
probability p, and Paul must determine the unknown number 2 correctly with probability at 
least 1 6, for a given 6 > 0. In this chapter we give a reduction to show that searching in 
the probabilistic error model is no more difficult than searching in the linearly bounded model. 
Formally, we show that if A; is an algorithm which, given n and r, solves the linearly bounded 
error problem in f(n,r) questions, then we can construct an algorithm A, which solves the 
probabilistic error problem in f(en, +22) questions where c is a constant depending on p and 6. 
An O(log n) bound for the probabilistic error model with membership questions follows easily 


from the results of the previous chapter. We also generalize our results to the unbounded 


domain. 


9.1 The Reduction 


The terms “algorithm” and “strategy” will be used somewhat interchangeably, since a winning 
strategy for the Pusher in the Chip Game corresponds to an algorithm to solve the search 
problem with errors, and vice versa. 

We now construct an algorithm which solves the probabilistic error problem from an algo- 
rithm which solves the linearly bounded error problem. Let A, be an algorithm which solves 


the linearly bounded error problem. A, requires values for n and r, as well as access to an 
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oracle whose errors are linearly bounded (an oracle which gives at most ri errors to any initial 
sequence of 2 questions). Algorithm A, will ask f(n,r) questions and will return the correct 
element x € {1,...,n} with certainty. Let A, be an algorithm which solves the probabilistic 
error problem. A, requires values for n, p and é, as well as access to an oracle whose errors are 
random (an oracle which lies randomly and independently with probability p). Algorithm A, 
will ask g(n, p, 6) questions and will return the correct element « € {1,...,n} with probability 
at least 1 — é. 

In order to solve a probabilistic error problem with a linearly bounded error algorithm, we 
must insure that the errors made by the probabilistic oracle fall within those allowed by the 
linearly bounded error algorithm (with high probability). One method to accomplish this is to 
set r > p. This assures that in the long run, with high probability, the number of lies told by the 
probabilistic oracle will be fewer than the number of lies the linearly bounded error algorithm 
can tolerate. The danger here lies at the beginning of the game where it is relatively likely 
that too many lies will be told, and hence the correct chip will be thrown out by the linearly 
bounded error algorithm. To overcome this difficulty, we must prevent the linearly bounded 
error algorithm from throwing out the correct chip in this critical stage. 

We proceed in two stages. In the first stage, we play a modified game with excess chips in 
such a way as to guarantee that the correct chip is not thrown out until at least m questions 
have been asked of the probabilistic oracle. In the second stage, we find the correct chip among 


those remaining with high probability. 


9.1.1 Stage 1 


We begin the game by setting r midway between p and 1/2. Thus, r = iitte = oe This 
insures that the number of errors given by the probabilistic oracle will be fewer than the number 
of errors which can be tolerated by the linearly bounded error algorithm in the long run with 
high probability. To insure that the correct chip does not cross the boundary line before the 


m 


m-th step, we begin the game with n2 chips. 


In the first critical — m steps, we intercept algorithm A,’s queries to the oracle and answer 


them so as to maximize the number of chips which are left at level 0. We first note that after 


these —"m steps, the boundary line will be at (1 —r)m. Second, since the number of chips at 
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(1-r)m 


n2°* n 
(1-r)m/r_ steps m steps 
O° 22 Bi es OF 28. 232 (1-r)ym 0. De 2y Beas (1-1)m m 
~<— We answer questions ——> <— Oracle answers questions ——» 


Figure 9.1: Stages of the Reduction 


level 0 are reduced by at most half in each step, there will be at least n chips remaining after 


these +m steps. See Figure 9.1. 


r 


9.1.2 Stage 2 


i=, 


Associated with each chip in the current game is an element of the set {1,...,n27™}. For 


a chip u, let this be the OldValue(u). We now establish a new correspondence between n 
of the remaining chips at level 0 and the set {1,...,n}. For a chip u, this will be New- 
Value(u). This new correspondence is order-preserving in the following sense: for chips u 
and v, OldValue(u) < OldValue(v) iff NewValue(u) < NewValue(v). The necessity for estab- 
lishing an order-preserving correspondence stems from the need to have this reduction apply to 
the searching problem where only comparison questions are allowed. We now continue running 
algorithm .A,, sending his queries to the probabilistic oracle after translating them thus: Let 
C = {u: A; picks u and NewValue(u) € {1,...,n}}. That is, C is the set of selected chips 
which have defined NewValues. Let Sc = {NewValue(u) : u € C}, that is, Sc is the set of 
associated New Values. If Se # 0, then we ask the probabilistic oracle about S¢ and return the 
oracle’s answer to A,. If S¢ = 9, then we could ourselves immediately answer “no”. However, 
it is more convenient in the analysis to have the probabilistic oracle answer all questions in 
this stage. Thus, when Sc = 9, we ask the probabilistic oracle about {1,...,2}, and return 
the opposite of its answer to A,. Suppose that A; finishes and returns chip u. We then return 


New Value(u) or “fail” if New Value(u) is not defined. 
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9.1.3 The Analysis 


We now claim that for an appropriate m, the above procedure will terminate with the correct 
value with probability at least 1 — 6. We in fact show that the probability that the “correct 
chip” ever crosses the boundary line is at most 6. If the correct chip never crosses the boundary 
line, then the linearly bounded error algorithm must return the correct chip when it terminates, 
and hence the correct answer will be obtained. 

Our analysis makes use of Hoeffding’s Inequality [18] to approximate the tail of a binomial 
distribution. Let GE(p,m,n) be the probability of at least n successes in m Bernoulli trials, 
where each trial has probability of success p. Hoeffding’s Inequality can then be stated as 
follows: 


GE(p,m,(p+ a)m) < e?"™ 


After the first "tm steps, the correct chip will be at level 0, and the boundary line will be 
at level (1 — r)m (see Figure 9.1). Since a chip can move at most one level per question and 
the boundary line moves at a rate r, none of the n remaining chips at level 0 will cross the 
boundary line until at least m questions have been asked of the probabilistic oracle. For any 
j > m, the probability that the correct chip is past the boundary line after 7 questions have 
been asked of the oracle is given by GE(p,j,jr+(1—1)m). The probability that the correct 


chip is ever past the boundary line is therefore at most 


> GE(p.jnjr-+ (1—r)m). 


j=m 
Given that: 


e Ifn>n’ then GE(p,m,n) < GE(p,m,n’) 


we can apply Hoeffding’s Inequality: 


Me 


3° GE(p.j.jr+(1— rm) < 


j=m 


GE(p,j,J7) 


m 


we. 
U 


l| 
Me 


GE(p, j,(p+ *)s) 


mT 
3 
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< ens 


oO 
jo=m 


Since we would like this sum to be at most 6, we can now solve for m. Let y = e-2-*)" | Note 


that y < 1. 
Viv < 6 
j=m 
J << 6 
1l-y 7 
1 1 
m In; +In-> 
In+ 
5 
For y = e722) = e- 3 we obtain 


_ (1=2p)? 
my > ols tS inll/C - e 3 5 


- (1 — 2p)? 
Noting that for0< p< : 0< G=2p)" < :; we can use the fact that forO <2 < :; 08 > = 
to pick 
1 128/15 
8 ln s+ 8 ln Gp 
m= oO 
(1 — 2p)? 


We can now conclude the following theorem: 


Theorem 28 Let A; be a linearly bounded error algorithm which requires f(n,r) questions. 
Then A, can be used to solve a probabilistic error problem specified by n, p, and é in f(en, +22) 


questions where c = 21?” and m is as given above. 


An O(logn) bound for the probabilistic error model with membership questions now easily 


follows from the results of the previous section. 


Theorem 29 The problem of searching for an unknown element x € {1,...,n} with confidence 
probability 6 in the presence of random errors (occurring randomly and independently with fixed 


probability p < 1/2) can be solved with O(log n) membership questions. The dependence of these 


bounds on p and 6 is polynomial in a and logarithmic in 1/6. 
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9.2 The Unbounded Domain 


We now consider the problem of searching for an unknown integer in the presence of random 
errors where no bound on the unknown number is known. Let this unknown integer be n. Our 
strategy proceeds in two stages. In the first stage, we obtain a bound for the integer n. In the 
second stage, we apply our techniques for searching in the bounded domain given above. To 
insure that our overall procedure fails with probability at most 6, we require that each of these 


two stages fails with probability at most 6’ = 6/2. 


9.2.1 Stage 1 


By obtaining the correct answers to [lglgn] questions of the form “Is « < 2?'?” as in sec- 
tion 8.4.1, we can bound the unknown number n by at most n?. 

We might now imagine determining the correct answers to these [lglg] questions by asking 
each one sufficiently often so that majority is incorrect with some sufficiently small probability. 
Unfortunately, to determine how much error is “sufficiently small” requires that we know the 
value of n. Since n is unknown here, we will require a more subtle querying algorithm. 

To insure that our procedure fails with probability at most 6’, we require that the correct 
answer to question tis obtained with error at most 6’/2'. Consider asking the i-th question m(?) 
consecutive times and taking the majority vote of the responses to be the “correct” answer. 


The probability that our posited answer is incorrect can be calculated as follows: 


Pr[majority vote is wrong] =  Priat least half errors] 
GE(p, mi), m(3)/2) 


GE(p, m(2), (p+ [1/2 — p]) - m(z)) 
<7 21/2-p)*m(i) 


(1=2p)? m(i) 
= € 2 


Since we require this probability to be at most 6’/2', we obtain the following: 


(1=2p)? m(i) 
2 


€ < 6/2! 
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In 4 


mi) > a le = + j 


Now, since our procedure will terminate (with probability at least 1 — 6’) after the correct 


answers to [lglg] questions have been obtained, we arrive at an overall question bound of 


Ss figlgn| In4 1 
mc = fg 
i=l tl (1 ~ 2p)? 6 
In 4 1 fIglgn](flglgn] + 1) 
= —— |] [lel lg —- 4+. 
= O([lglgn]*) 


We thus bound the unknown number n by at most n? using O([lglg n]*) (comparison) questions. 


9.2.2 Stage 2 


We can now simply apply the bounded searching techniques for membership questions described 
in previous section or the bounds of Feige et al. [12] for comparison questions. We can thus 
obtain the correct answer (with high probability) in an additional O(lg n”) = O(lg n) comparison 


or membership questions. Thus, we can conclude the following theorem: 


Theorem 30 The problem of searching for an unknown element n in the unbounded domain 
of all positive integers with confidence probability 6 in the presence of random errors (occur- 
ring independently with fixed probability p < 1/2) can be solved with O(lgn) comparison or 
membership questions. The dependence of these bounds on p and 6 is polynomial in a and 


logarithmic in 1/6. 


CHAPTER 10 


Conclusions and Open Questions 


We have examined the problem of searching in a discrete domain under two different error 
models: the linearly bounded error model and the probabilistic error model. 

In the linearly bounded error model, we have shown that O(lgn) membership questions are 
sufficient to search in both the bounded and unbounded domains. With comparison questions, 


1 


) and O([nlg* n]'8=*) in the bounded and unbounded domains, 


1 
Pr 


we show bounds of O(n'8 = 
respectively. 

Our reduction from the probabilistic to the linearly bounded error model shows that the 
searching problem is at least as difficult to solve in the linearly bounded error model as in the 
probabilistic error model. This gives evidence that the linearly bounded error model deserves 
further investigation. A corollary of this reduction gives another proof of the O(lgn) bound on 
membership questions required to search with probabilistic errors. Previously known bounds 
are also extended to the unbounded domain. 


Two questions arise directly from this work: 


1. In the linearly bounded error model, can we show a logarithmic upper bound on the 
number of comparison questions required when the error rate is between 1/3 and 1/2? 
Using techniques similar to ours, Borgstrom and Kosaraju [8] have recently shown that 


this is the case. 
2. Can a strict inequality be shown between the probabilistic and linearly bounded models 
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with respect to the problem of searching? That is, can it be shown that searching in the 
presence of linearly bounded errors with some question class requires an asymptotically 
greater number of questions than searching in the presence of random errors with the 


same question class? This problem remains open. 
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